Linux, Unix and BSD: Tips, Tricks, Useful Commands and Interesting ways to get things done! Share your knowledge!

[Blue-Vision]

You read the title: Post any and all useful, interesting and helpful commands, applications, and ways to get things done in an interesting or timely fashion!

[Blue-Vision]

I'll recap with the first of two things I've posted about before.

 

Zram: 

https://en.wikipedia.org/wiki/Zram

https://www.kernel.org/doc/Documentation/blockdev/zram.txt

http://askubuntu.com/questions/174579/how-do-i-use-zram

https://wiki.archlinux.org/index.php/Maximizing_performance#Zram_or_zswap

http://askubuntu.com/questions/471912/zram-vs-zswap-vs-zcache-ultimate-guide-when-to-use-which-one

 

Purpose: Allocate up to 50% of your physical RAM to act as 75% compressed SWAP space; ie. turns 4gb (2gb at a time) into 6gb of useable RAM, before using the Hard Disk Swap allocation (which is far slower). The performance gains are 10,000~50,000 times that of Hard Disk Swap since physical RAM is only limited by the speed of the system bus itself.

Pros: Lets any machine, even ones which don't necessarily need the extra RAM function as though they had additional RAM. Furthermore, since Read/Write lifetimes on SSD's are a concern, this technology limits that concern considerably as the system will rely on RAM Swap first before going to Disk Swap.

Cons: Needs a 3-10% CPU overhead for the decompression routines. When RAM intensive operations are going on (such as gaming, or workloads that are especially heavy), that extra 3-10% overhead can slow things down a bit. Newer Processors will barely notice this overhead, however; even the so-called "budget" ones, as they employ a multi-thread, multi-process design.

Maturity: Quite mature and stable. Was included into the Linux Kernel as of version 3.14, March 2014. The technology is continually being improved, and compression algorithms are likewise under constant testing and development.

 

How do I use this? Glad you asked. Most Linux Distributions come with it built-in, but not activated. You can type:

cat /proc/swaps

to see your current swapspace use.

 

To activate Zram on Ubuntu-based Distros (including Mint), you need to install zram-config.

sudo apt-get install zram-config

It'll activate automatically after it's installed. Type the command cat /proc/swaps again, and you'll see *this* (or something similar to mine):

 

 

And that's that! Read the documentation on how to tweak your Zram swaps.

 

Enjoy!

 

Edited December 20, 2015 by Unobscured Vision
Did I have a stroke?? Yikes ..

[Circaflex]

[Blue-Vision]

Alias:

https://en.wikipedia.org/wiki/Alias_(command)

http://www.linfo.org/alias.html

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=linux%20alias%20command

http://askubuntu.com/questions/1414/how-to-create-a-permanent-alias

 

Purpose: An 'alias' is a shortcut for the terminal. It lets the user assign an 'alias' for a long command string, saving the user a lot of time and potential problems from mis-typed commands. Instead of:

sudo apt-get update && sudo apt-get upgrade

you can make an alias command that will let you type

update

that will simply ask for your password (as the long command does) and away you go! You can do this with any command that you would use the terminal for, but practicality would normally suggest that you save aliases for things you use frequently.

 

How do I use this? There's the "approved and suggested" way, and the "definitely not approved and certainly not suggested" way (which worked for me). I'm going to give you the not approved way, and I might catch some flak for this; but if you're the only user on your machine, and behind a firewall (most routers have them) then you should be fine. It's considered a security risk to create aliases like this; but since most Linux Distributions have a number of system-wide aliases built in it should be fine.

 

As superuser, open /etc/bashrc.bash with a text editor and find this line:

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

Aliases need to be entered below this line. It's a good idea to back up this file!

 

Your aliases need to be in this format:

alias update='sudo apt-get update && sudo apt-get dist-upgrade'

Make as many as you like. Remember, anything you can do on a terminal line you can do in an alias. They can be as complex as you need them to be. Save the file when you're done.

 

A good example, as in my original post about aliases, was the commandline program dfc. I believe it was ported to Linux from BSD. It's got some great features, but you have to use the command switches. If you were to simply install it and type dfc, this is what you'd get:

But after, using the command switches -TasowW:

Then you alias it in /etc/bashrc.bash:

alias df='dfc -TasowW'

Now, whenever I type "df" in my terminal (without the quotes, of course) and my terminal window is wide enough, I get that nice, buff, dfc output.

 

Enjoy the power of alias, and use it wisely.

[Barney T. Administrators]

Pinned!

 

https://www.neowin.net/forum/topic/1282390-linux-unix-and-bsd-tips-tricks-useful-commands-and-interesting-ways-to-get-things-done-share-your-knowledge/

[Blue-Vision]

BASH General Reference and Scripting:

 

https://en.wikipedia.org/wiki/Bash_(Unix_shell)

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=Learn+bash

https://learnxinyminutes.com/docs/bash/

http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html

http://linuxcommand.org/learning_the_shell.php

http://wiki.bash-hackers.org/scripting/tutoriallist

 

Definition and Purpose: (From Wikipedia): "Bash is a Unix shell and command language written by Brian Fox for the GNU Project as a free software replacement for the Bourne shell. Released in 1989, it has been distributed widely as the shell for the GNU operating system and as a default shell on Linux and OS X. It has been ported to Microsoft Windows and distributed with Cygwin and MinGW, to DOS by the DJGPP project, to Novell NetWare and to Android via various terminal emulation applications.

 

Bash is a command processor that typically runs in a text window, where the user types commands that cause actions. Bash can also read commands from a file, called a script. Like all Unix shells, it supports filename globbing (wildcard matching), piping, here documents, command substitution, variables and control structures for condition-testing and iteration. The keywords, syntax and other basic features of the language were all copied from sh. Other features, e.g., history, were copied from csh and ksh. Bash is a POSIX shell, but with a number of extensions.

 

The name itself is an acronym, a pun, and a description. As an acronym, it stands for Bourne-again shell, referring to its objective as a free replacement for the Bourne shell. As a pun, it expressed that objective in a phrase that sounds similar to born again, a term for spiritual rebirth. The name is also descriptive of what it did, bashing together the features of sh, csh, and ksh." (annotations removed)

 

Bash Scripts -- Purpose and Howto: 

(reference: https://en.wikipedia.org/wiki/Bash_(Unix_shell)#Features)

(from Wikipedia): "The Bash command syntax is a superset of the Bourne shell command syntax. Bash can execute the vast majority of Bourne shell scripts without modification, with the exception of Bourne shell scripts stumbling into fringe syntax behavior interpreted differently in Bash or attempting to run a system command matching a newer Bash builtin, etc. Bash command syntax includes ideas drawn from the Korn shell (ksh) and the C shell (csh) such as command line editing, command history, the directory stack, the $RANDOM and $PPID variables, and POSIX command substitution syntax $(…).

 

When a user presses the tab key within an interactive command-shell, Bash automatically uses command line completion to match partly typed program names, filenames and variable names. The Bash command-line completion system is very flexible and customizable, and is often packaged with functions that complete arguments and filenames for specific programs and tasks.

 

Bash's syntax has many extensions lacking in the Bourne shell. Bash can perform integer calculations ("arithmetic evaluation") without spawning external processes. It uses the ((…)) command and the $((…)) variable syntax for this purpose. Its syntax simplifies I/O redirection. For example, it can redirect standard output (stdout) and standard error (stderr) at the same time using the &> operator. This is simpler to type than the Bourne shell equivalent 'command > file 2>&1'. Bash supports process substitution using the <(command) and >(command) syntax, which substitutes the output of (or input to) a command where a filename is normally used. (This is implemented through /proc/fd/ unnamed pipes on systems which support that, or via temporary named pipes where necessary).

 

When using the 'function' keyword, Bash function declarations are not compatible with Bourne/Korn/POSIX scripts (the Korn shell has the same problem when using 'function'), but Bash accepts the same function declaration syntax as the Bourne and Korn shells, and is POSIX-conformant. Because of these and other differences, Bash shell scripts are rarely runnable under the Bourne or Korn shell interpreters unless deliberately written with that compatibility in mind, which is becoming less common as Linux becomes more widespread. But in POSIX mode, Bash conformance with POSIX is nearly perfect.

 

Bash supports here documents. Since version 2.05b Bash can redirect standard input (stdin) from a "here string" using the <<< operator.

 

Bash 3.0 supports in-process regular expression matching using a syntax reminiscent of Perl.

 

Bash 4.0 introduced support for associative arrays. Associative arrays allow a fake support for multi-dimensional (indexed) arrays, in a similar way to AWK:

$ declare -A a         # declare an associative array 'a' faking a bi-dimensional indexed array
$ i=1; j=2             # initialize some indices
$ a[$i,$j]=5           # associate value "5" to key "$i,$j" (i.e. "1,2")
$ echo ${a[$i,$j]}     # print the stored value at key "$i,$j"
5

Brace expansion

Brace expansion, also called alternation, is a feature copied from the C shell. It generates a set of alternative combinations. Generated results need not exist as files. The results of each expanded string are not sorted and left to right order is preserved:

$ echo a{p,c,d,b}e 
ape ace ade abe
$ echo {a,b,c}{d,e,f} 
ad ae af bd be bf cd ce cf

Users should not use brace expansions in portable shell scripts, because the Bourne shell does not produce the same output.

#!/bin/sh

# A traditional shell does not produce the same output
echo a{p,c,d,b}e # a{p,c,d,b}e

When brace expansion is combined with wildcards, the braces are expanded first, and then the resulting wildcards are substituted normally. Hence, a listing of JPEG and PNG images in the current directory could be obtained using:

ls *.{jpg,jpeg,png}    # expands to *.jpg *.jpeg *.png - after which,
                       # the wildcards are processed
echo *.{png,jp{e,}g}   # echo just show the expansions -
                       # and braces in braces are possible.

In addition to alternation, brace expansion can be used for sequential ranges between two integers or characters separated by double dots. Newer versions of Bash allow a third integer to specify the increment.

$ echo {1..10}
1 2 3 4 5 6 7 8 9 10
$ echo file{1..4}.txt
file1.txt file2.txt file3.txt file4.txt
$ echo {a..e}
a b c d e
$ echo {1..10..3}
1 4 7 10
$ echo {a..j..3}
a d g j

When brace expansion is combined with variable expansion the variable expansion is performed after the brace expansion, which in some cases may necessitate the use of the built-in eval function thus:

$ start=1; end=10
$ echo {$start..$end}  # fails to expand due to the evaluation order
{1..10}
$ eval echo {$start..$end} # variable expansion occurs then resulting string is evaluated
1 2 3 4 5 6 7 8 9 10

(and the wikipedia article continues for a LONG time ... quoting the entire thing will violate the T&C of "Fair Use").

 

How do I use this? Read the links. There's FAR too much that bash scripting can do. It literally IS like learning to use another programming language!

 

Enjoy!

[Blue-Vision]

New Year, new tip. Did you know about this one? I didn't either, until yesterday.  

 

Defragmenting an Online (in-use!) Ext4 Filesystem:

 

http://askubuntu.com/questions/221079/how-to-defrag-an-ext4-filesystem

http://superuser.com/questions/536788/do-ext4-filesystems-need-to-be-defragmented

http://manpages.ubuntu.com/manpages/wily/man8/e4defrag.8.html

 

Purpose: Over time, every filesystem (even ones like Ext2/3/4) will fragment. Much more well-known NTFS and FAT32-f ("Fast-FAT") filesystems fragment at a faster rate, as is well-documented, and many utilities have been developed to deal with it. On Linux, the Ext(x) Journaling filesystems were developed to minimize fragmentation and filesystem error as much as possible but it still happens and tools were needed to deal with it. The currently preferred tool (e4defrag) matching up to the currently preferred filesystem (Ext4) has been available to Ubuntu-based Distributions since Ubuntu 12.04 LTS (April 2012) and was improved upon in the .3 release. It was ported to the e2fsprogs package set and is available for nearly all modern Linux and Unix distributions now.

 

How do I use this? It's commandline-driven, so off to our Terminal. It's not as difficult as it may appear on the surface -- in fact, it's downright easy to use.

 

 

Typing e4defrag --help (something everyone should do when trying out a new or unfamiliar program!) shows us our application switches. As you can see, there's not a whole heck of a lot that this one says. One of the links I provided at the top of the post is also for the manpage to the application, which will help us out a great deal. Take a look at that in another tab, then come back.

 

--------------------------------------------------

 

Now that you've read the manpage (also known as a Manual Page), let's get started. Back to the Terminal window!

 

Running e4defrag in its' default condition will not do much, as the manpage says. You'll be able to defrag your /home directory and any other directories you own, but that's it. Remember, this is 'nix. Security. We can't just muck about the system, you know -- even when we're doing maintenance.

 

Let's turn up the volume. Our command is going to be:

 

sudo e4defrag -c /home/[your folder]

Why are we doing this as sudo? Otherwise we won't be able to see the fragmentation count afterwards. And since we're only doing this on your /home directory (the directory of likeliest fragmentation), the time to complete the operation is significantly reduced.

 

** Performing a sudo e4defrag / will defrag the entire disk. And WILL take all night. I found out the hard way ....  So don't do it unless you mean to do it. **

 

Now then.

 

 

As you can see, I've got no issues on my disk because of the FULL defrag I did yesterday ... your results may vary. Scan times will also vary but should take no longer than 5 seconds per 30,000 extents. If you've got a lot of data in your /home directory, expect the -c (check) scan to take upwards of two or three minutes.

 

If you need to defragment your /home directory, you may do so by invoking sudo e4defrag /home/[your folder] . If you'd like a detailed readout while that defrag is going on, use the -v switch before the directory (so that it becomes sudo e4defrag -v /home/[your folder] .

 

Hope this helps! For future reference, and ease/speed, consider creating Alias(-es) to scan and another to actually defrag your /home directory, and even another set for the entire drive!  

Edited January 2, 2016 by Unobscured Vision

[binaryzero]

This is hardcore. 

 

[BinaryData]

And several things just helped me out majorly.

 

@Unobscured VisionThank you for this!!!!!!

[Blue-Vision]

51 minutes ago, BinaryData said:

And several things just helped me out majorly.

 

@Unobscured VisionThank you for this!!!!!!

You're quite welcome. That's what I'm here for!  

[Dhillon]

I have recently switched from windows to linux (mint). Can anyone suggest an image editor for light editing/resizing and ability to export 8 bit png images for web use.

 

I am long time fireworks user & gimp is too advance for me.

[Blue-Vision]

Simple Image Editors for Linux/Unix:

 

http://askubuntu.com/questions/164473/simple-image-editor

http://codecondo.com/top-photo-editors-for-linux/

 

There you go, Dhillon.  

[simplezz]

Because I see people struggle with this problem so often:

 

Restoring A Distro's Grub After It Gets Overwritten

 

It's quite common to see this kind of thing happen after installing another distro or Windows in dual/multi-boot setup. This should work on most systems.

 

1) First download a live cd ISO (preferably the same as the distro you're restoring) and write to a disc / usb pen drive. DD is the quickest way :-

dd if=LiveCD.iso of=/dev/sdd bs=4M

Be careful with that command and make sure the of= is either your cd (sr0) or a usb pen drive (dmesg | tail after inserting to confirm the device name). LiveCD.iso and sdd should be replaced with the iso you downloaded and the disc/pen device you're using respectively.

 

2) Reboot from disc/pen drive and enter live environment.

 

3) Chroot and restore grub :-

# replace sda1 with your distro's partition name. blkid / lsblk -f to get a listing.
mount /dev/sda1 /mnt

# setup file system linkages
mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev

chroot /mnt /bin/bash
grub-install /dev/sda

 

[Blue-Vision]

Thanks, @simplezz. I have run into this problem myself on occasion.

 

There's a GUI-based tool out there that will do the job called Boot-Repair, and it's installable during LiveCD/LiveUSB mode. I've used it before, and while it is useful and can help those who are not well-versed get up and running, I've also hosed my Windows boot loader using it. And, since it (by default) submits diagnostic data about your device unless you opt out of it, it isn't really recommended by the Community at large to use it.

[Ulyses]

2 hours ago, simplezz said:

Because I see people struggle with this problem so often:

 

Restoring A Distro's Grub After It Gets Overwritten

 

3) Chroot and restore grub :-

# replace sda1 with your distro's partition name. blkid / lsblk -f to get a listing.
mount /dev/sda1 /mnt

# setup file system linkages
mount -t proc none /mnt/proc
mount -o bind /dev /mnt/dev

chroot /mnt /bin/bash
grub-install /dev/sda

update-grub...?

[Blue-Vision]

Mount a local Hard Disk/Partition at Login instead of at Startup:

 

https://help.ubuntu.com/community/AutomaticallyMountPartitions

http://askubuntu.com/questions/83461/automatically-mount-ntfs-drive-when-i-login

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=mount%20a%20local%20hard%20disk%20at%20user%20login

 

Purpose: The standard interface for automatic handling of system-available Partitions and Hard Disks beyond the installation partition has typically been handled by "Disks", also called gnome-disks. This application does a decent job most of the time. The user fires it up, sets the desired mounting options for the various additional partitions and local hard disks, commits the changes, and presto. Usually there are no problems ...

 

Sometimes there are problems. For example, the user has decided they no longer need one of those partitions, formats it and forgets about the automount. Or, one of those local hard disks has failed silently. Perhaps one of those local hard disks is attached to the system in such a way that the system cannot possibly boot from it (at the BIOS/UEFI level) but is accessible after the system is up and running. If that entry is still present in the fstab file, the system will halt. The only way to fix it is either boot into Recovery Mode and perform a complicated series of commands to gain read + write access to the disk (because it is locked until then), or boot into a LiveCD, mount the system partition and repair fstab that way ... or even from Windows using the ext2/3/4 FS driver (the horror ....).

 

This tip will prevent the above from ever happening.  And it's a generally preferred and "safer" way to handle local mounts.

 

How do I use this? It's straightforward enough. Once you've done it a couple of times it won't seem so "daunting".

 

Step 1) We need to find out the UUID (Unique Disk Identifier) of the partition or Hard Disk you want to mount at login. You can choose to use the Terminal or Disks. For speed, we'll use the Terminal. Fire up your Terminal, and type:

ls -al /dev/disk/by-uuid/

You'll see a readout of your partitions and local Hard Disks. (If you've enabled Zram swaps, you'll see those in the list too.) If you know the partition you want (sda2, for example, will be my Windows installation), make a mental note that you see it; otherwise fire up Disks and find it. It's a good idea to keep the terminal window open for now. Don't do anything with Disks other than poke around. We're making sure that you know your partition/hard disk layouts. Once you have things sorted (and yes, you can get the UUID of the partition/local disk from Disks, but it's faster in the Terminal), proceed to Step 2.

 

Step 2) In your Desktop Environment, you should have a way to modify Startup Applications. Proceed to that tool directly.

 

Step 3) We're making a new Startup Application. Yes, it's 'Custom Command'. Name it whatever you want -- call it Nina, I don't care.  Our commandline will be:

udisksctl mount --block-device /dev/disk/by-uuid/<uuid>

Change the <uuid> with the series of letters and numbers that goes with your partition or hard disk. For example:

 

 

Will become ...

udisksctl mount --block-device /dev/disk/by-uuid/84AC8188AC817608

See how that works?

 

Now there's no worry that losing a partition or a local Hard Disk will keep your system from loading fully.  Enjoy!

[Gerowen]

I'm only here for a second and haven't read all of the replies, but I'll share some links to the two wiki articles I've written.

 

Recording from an audio loopback device (handy for screencasts and such)

https://wiki.debian.org/audio-loopback/

 

Using apt-file to figure out which packages (even ones that aren't installed) contain a particular file.

https://wiki.debian.org/apt-file/

[simplezz]

20 hours ago, Ulyses said:

update-grub...?

That's a stub for grub-mkconfig -o /boot/grub/grub.cfg. It only generates the config as far as I'm aware, it doesn't actually install grub in the MBR.

[Ulyses]

46 minutes ago, simplezz said:

That's a stub for grub-mkconfig -o /boot/grub/grub.cfg. It only generates the config as far as I'm aware, it doesn't actually install grub in the MBR.

I meant it's missing. It's probing for OSs and generates the menu.

Edited January 6, 2016 by Ulyses

[simplezz]

30 minutes ago, Ulyses said:

I meant it's missing. It's probing for OSs and generates the menu.

I see what you mean now. You're right, it's needed to pick up additional OS' added since it was last generated. That or manually running grub-mkconfig -o /boot/grub/grub.cfg.

[+John Teacake MVC]

Spoiler

This is awesome. 

 

[simonlang]

On 1/2/2016 at 6:54 AM, Unobscured Vision said:

New Year, new tip. Did you know about this one? I didn't either, until yesterday.  

 

Defragmenting an Online (in-use!) Ext4 Filesystem:

 

 

 

 

 

@Unobscured Vision  great tip thanks!  looks like my home directory is pretty fine

 

but good to know for the future! 

[Blue-Vision]

38 minutes ago, Simon Lang 9047 said:

 

 

@Unobscured Vision  great tip thanks!  looks like my home directory is pretty fine

 

but good to know for the future! 

Yep, it's good to run it every month or so. I've set up aliases for specific defragmentation needs myself: checkhome (and defraghome), then checksystem (and defragsystem).  

[simonlang]

ok here comes a tip, not that advanced, but i think good to know nevertheless:

 

- unhappy with the look of your system fonts in arch? >> try infinality-bundle!

 

In order to use the repository, please add the following to your pacman.conf:

[infinality-bundle]
Server = http://bohoomil.com/repo/$arch

If you want to access multilib libraries in x86_64 architecture as well, add also this:

[infinality-bundle-multilib]
Server = http://bohoomil.com/repo/multilib/$arch

Additionaly, if you want to use a comprehensive collection of free fonts from the infinality-bundle-fonts repository, append

[infinality-bundle-fonts]
Server = http://bohoomil.com/repo/fonts

as well.

Next, import and sign the key:

# pacman-key -r 962DDE58
# pacman-key --lsign-key 962DDE58

Refresh the package list and upgrade the system if necessary:

# pacman -Syyu

Finally, install the bundle with

# pacman -S infinality-bundle

for the minimal variant or, if you prefer a more complex and feature-rich solution, add any of infinality-bundle-multilib,ibfonts-meta-base and ibfonts-meta-extended (for the optional font collection). (Note that if you choose ibfonts-meta-extended, the base set of packages will be pulled automatically as its dependencies.)

 

this will result in very clean font style like seen in ubuntu.

[+Gary7 Subscriber²]

I should have posted this Here. Some great info.

 

http://www.brunolinux.com/

http://www.brunolinux.com/03-Installing_Software/Installing_Software_RPM.html

http://www.brunolinux.com/04-The_File_System/Partitions.html

http://www.brunolinux.com/05-Configuring_Your_System/Cron-Jobs_and_the_Cron_Daemon.html