Console magic. We select useful utilities for working in the terminal.

Mutt

Professional
Messages
1,056
Reaction score
644
Points
113
"Is this a matrix?!" - someone from your acquaintances will exclaim, looking over your shoulder at the laptop screen. No, not a matrix, you just frantically type something in the terminal. In this article I will show you more than two dozen utilities that will help you work with the Linux command line more efficiently, pleasantly and even beautifully.

SYSTEM

Pueue

The Pueue utility is an interesting thing for running long tasks and for working with the created task queue in the system, of course. It is needed for those who are quite obvious for such cases jobs/ fg/ bg/ screen/ tmuxfor some reason is not enough.

As the developers themselves write, Pueue is needed when tasks are very long or they need to be performed in parallel, while having full control over them. Personally, the utility seemed to me much more friendly than the standard bg/ fg.

INFO
Crongo is another attempt at a handy cron tool. The utility does about the same thing as Pueue, so it can be an alternative in a sense.

NQ
Nq is another simple utility that allows you to run queues from jobs on the system. Not that it was impossible to live without her, but sometimes it helps a lot. It looks much less attractive than Pueue, but it is also easier to work with.

It is so simple that you just need to see an example to use it:
Code:
% mkdir -p /tmp/downloads
% alias qget='NQDIR=/tmp/downloads nq wget'
% alias qwait='NQDIR=/tmp/downloads fq -q'
window1% qget http: //mymirror/big1.iso
window2% qget http: //mymirror/big2.iso
window3% qget http: //mymirror/big3.iso
% qwait

... wait for all downloads to finish ...

Vizex
If you work in the terminal for more than a week, you, of course, know about dfits dry output. Let's decorate it and make it closer to people!
Vizex and vizexdf are an upgrade dfthat performs its task much more clearly and externally.

egaxLCCOSWY.jpg

What vizex looks like

bashtop
In hacker movies, people see huge pivot tables with system information and graphs, but you still don't have them? No problem! Catch bashtop - this is just such a dashboard. If you suddenly have a free monitor on which you can hang it, you know how to add +1000 to your hype in the eyes of your visitors.

hBFMuXAQHIc.jpg

Works!

Bashtop is able to display not only general information, but also details for each process. Allows these processes to be sorted by various parameters and is easily configurable.

And there is also a Python implementation of this handsome man - bpytop. It looks no less impressive, and does not lag behind in terms of capabilities.

Rhit
An amusing console utility for analyzing nginx logs. Meet Rhit - this, of course, is not GoAccess, but it looks interesting too.

Knows how to draw graphs of the request rate directly in the console.
qjyYD68acfY.jpg
qjyYD68acfY.jpg

Screenshots are stepped from the program website

There is an analysis of trends in queries and a convenient graphical output of this to the console.
5a1UjX-kml0.jpg


Of course, everything can be filtered to keep track of only what is needed.

Lnav
Lnav is a log analyzer that can work not only with nginx, unlike Rhit. Here is a list of its merits:
  • can bring all required logs to one screen. You can set monitoring of logs of several services at once, and it will show everything at once;
  • can highlight text by format. There are ten built-in formats, including one "general", that is, suitable for almost any log;
  • automatically detects compressed logs in gzip and bzip2 formats and decompresses them on the fly;
  • knows how to filter based on regular expressions. If you display a lot of logs at once, you can filter out unnecessary ones;
  • can build histograms of messages by time;
  • outputs XML and JSON nicely. Just press Shift + P;
  • the logs can be accessed as a virtual SQLite database, which is updated along with the logs in real time;
  • lnav supports different themes for its interface;
  • when entering commands, there is also syntax highlighting and auto-completion.
w5MdP3npFcw.jpg


lnav parses sshd logs
Directly on the sci-ones have ready binaries for Linux and MacOS: apparently, this is in case your server - old MacBook.

Butterfly Backup
Butterfly Backup is such an interesting wrapper around rsync that can create and restore backups. The list of skills is really impressive:
  • silent backup support;
  • all backups are neatly sorted;
  • viewing information about a specific backup;
  • support for different copy modes;
  • you can back up several computers at once, and in parallel;
  • you can restore a copy even on a computer other than the one where it was created. In theory, this can be useful when migrating to new hardware. Moreover, you can restore a backup even on a different operating system;
  • policies regarding old backups are supported: they can be deleted when the limit of the occupied storage is reached;
  • export of backups for convenient transportation by third-party means.

You can install Butterfly Backup in just three commands:
Code:
git clone https://github.com/MatteoGuadrini/Butterfly-Backup.git
cd Butterfly-Backup
sudo python3 setup.py

An example of use from the author of the utility:
Code:
# Full copy
bb backup --computer pc1 --destination /nas/mybackup --data User Config --type MacOS --mode Full
# Incremental copy
bb backup --computer pc1 --destination /nas/mybackup --data User Config --type MacOS

If you look at the list of copies with the command bb list --catalog /nas/mybackup, you can see a brief description of each backup taken.

BUTTERFLY BACKUP CATALOG
Code:
Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-03 17:50:36
Backup id: 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-06 07:26:46
Backup id: cc6e2744-9944-11e8-b82a-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-06 08:49:00

Some important details are missing here, so let's take a closer look at one of them.
Code:
bb list --catalog / nas / mybackup --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0
Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Type: Full
Timestamp: 2018-08-03 17:50:36
Start: 2018-08-03 17:50:36
Finish: 2018-08-03 18:02:32
OS: MacOS
ExitCode: 0
Path: / nas / mybackup / pc1 / 2018_08_03__17_50
List: etc
Users

Finally, recovery:
Code:
bb restore --computer pc1 --catalog / nas / mybackup --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0

SCRIPTS
Bash bible

For beginners and intermediate residents of the console, there is the Bash Bible service (and its sibling sh bible): there are examples of implementing various tasks in scripts written exclusively in pure Bash (or sh).

In order not to go far - here are a couple of examples of traversing all files and folders in a directory in pure Bash without using ls.
Code:
# Greedy example.
for file in *; do
printf '% s \ n' "$ file"
done
# PNG files in dir.
for file in ~ / Pictures / *. png; do
printf '% s \ n' "$ file"
done
# Iterate over directories.
for dir in ~ / Downloads / * /; do
printf '% s \ n' "$ dir"
done
# Brace Expansion.
for file in / path / to / parentdir / {file1, file2, subdir / file3}; do
printf '% s \ n' "$ file"
done
# Iterate recursively.
shopt -s globstar
for file in ~ / Pictures / ** / *; do
printf '% s \ n' "$ file"
done
shopt -u globstar

INFO
It would be wrong not to mention the sacred zsh with the oh-my-zsh plugin and custom themes. Nothing more convenient for the terminal has been invented.

Useful one-liners
I have in my bookmarks an interesting resource with a variety of one-liners - a kind of library of commands for all occasions.

For example, here's how to mount an NTFS partition from a VirtualBox virtual disk (VDI):
Code:
$ mount -t ntfs-3g -o ro, loop, uid = user, gid = group, umask = 0007, fmask = 0117, offset = 0x $ (hd -n 1000000 IMAGE.vdi | grep "eb 52 90 4e 54 46 53 "| cut -c 1-8) IMAGE.vdi / mnt / vdi-ntfs

Conty
If you constantly work in the terminal, sometimes you need isolated sandboxes. Some do it through Docker, creating an OS container and copying program files back and forth; others create and work in a chroot environment. Both methods require a lot of steps and root, so they are inconvenient to use.

There is an exit! The Conty tool makes launching the sandbox easy and fast. No root is required to run the container, and the files are saved after exiting the sandbox.

There are several advantages:
  • just one executable file. No strapping - downloaded and working;
  • compiled on the basis of Arch Linux, that is, it immediately contains the latest driver versions;
  • does not require root rights to run;
  • contains Vulkan and OpenGL, that is, it is suitable for games;
  • works without an overhead - this is not a virtual machine for you;
  • isolates the host file system, but provides a convenient sandbox-to-host interaction.
To work needed just anything: tar, fuse2, coreutilsand bash.

An example of launching from the author:
Code:
./conty.sh steam
./conty.sh lutris
./conty.sh playonlinux
./conty.sh wine app.exe

NET

SX Network Scanner

According to the developers, SX Scanner is almost the best alternative to Nmap. Here is a list of suggested features:
  • 30 times faster than Nmap;
  • ARP scanning to find live hosts on local networks;
  • ICMP scanning to detect firewall rules;
  • classic TCP SYN scan (scan using half-open connections) to find ports;
  • scanning to bypass some firewalls using TCP FIN, NULL and Xmas packets. Nmap can do the same, as you remember;
  • you can manually set any flags in the scan packages, and get the response flags in the report. Quite an interesting feature that I have not met anywhere else, but I have never needed it, to be honest;
  • even search for UDP ports. The reliability of the search, of course, leaves much to be desired, but the use of ICMP to refine the result is a definite plus;
  • scanning some applications: this includes Docker, Elasticsearch, and Socks5. For Docker, an open Docker API is looked for, through which information about the node is read; for Elasticsearch, information about the cluster and its indexes is dumped. With Socks5, everything is quite simple - sx only reports the fact that Socks5 is working without any details;
  • outputs the results in JSON - just like people! Of course, it is much more convenient for automatic processing.
Sx runs from the terminal (as, indeed, most of the programs reviewed today) and is written entirely in Go, which explains its performance.

Another beautiful instrument in your piggy bank "pseudographic svistoperdelok» - the ping from a visa-ali-zatsi-it directly in a terminal window.

grepcidr - parsing IP addresses
Ooty-lits for those who are already tired of writing regular season to search for the IP-addresses using grep. It works a little faster than it does grep, but in order to need a separate utility, you really have to have a lot of work with IP addresses.

sish
When you need to forward some service on the Internet without having an honest "white" IP, you can use services like Ngrok or Serveo. By and large, there are not many alternatives: bare SSH or artisanal self-hosted solutions.

Tunneling over SSH
You, of course, know about the possibility of forwarding ports through SSH. What about a full VPN? On this topic there is an excellent material on the Robotmoon, if you know English. However, the text is flavored with pictures with schemes of command work, and all examples can be painlessly run at home and tried in action.

The sish project also belongs to such homemade products - a beautiful and convenient alternative to Ngrok, built on top of regular SSH.

Sish isn't just a bunch of scripts to run SSH forwarding. This is a full-fledged implementation of an SSH server, which only knows forwarding, but it does it much better than the officially distributed version. In particular, sish can listen and multiplex HTTP and HTTPS traffic with WebSocket support (including encrypted) if you specify remote port 80 or 443 when starting forwarding. Otherwise, the TCP port will be forwarded if it is not busy on the server ...

The official installation instructions consist of just two commands:
Code:
docker pull antoniomika / sish: latest
docker run -itd --name sish \
  -v ~ / sish / ssl: / ssl \
  -v ~ / sish / keys: / keys \
  -v ~ / sish / pubkeys: / pubkeys \
  --net = host antoniomika / sish: latest \
  --ssh-address =: 22 \
  --http-address =: 80 \
  --https-address =: 443 \
  --https = true \
  --https-certificate-directory = / ssl \
  --authentication-keys-directory = / pubkeys \
  --private-key-location = / keys / ssh_key \
  --bind-random-ports = false

After that, you can simply forward ports as through regular SSH. But it could have been done this way, but you'd better see what happens if you make some settings according to the instructions in the repository:
Code:
ssh -R hacker_com: 80: localhost: 8080 your_domain.com

Op - and https://hacker_com.your_domain.comwill be transparently redirected to your local port 8080! Of course, as many sessions as you want can live at the same time.

Termshark
Termshark is a console frontend for tshark. Knows how, for example, to read pcap or capture traffic in real time, just like the "adult" Wireshark.

There is support for filters, you can automatically collect TCP streams from individual packets, view the contents of packets in accordance with the protocol.

Termshark is written in Go and is assembled for any platform - ready-made builds are already available for Linux, Windows, BSD, Android (on top of Termux) and even macOS!

SSH scripts
For SSH, we have collected several useful, and simply interesting tools at once.

To begin with - Uwe-house-ment in the Telegram when the SSH-login. A trifle, but nice!
nhPdjIhn_Fk.jpg


Installation is as simple as two rubles: download the script to /usr/local/bin/ssh2tg.sh(or another convenient folder), change the settings in the script (bot API token and chat ID to send notifications to), hang the execution attribute and add the /etc/pam.d/sshdfollowing line to the end:
Code:
session optional pam_exec.so type = open_session seteuid /usr/local/bin/ssh2tg.sh

The second one is to show you a little SSH Attack Stats agent, which will display an operational summary of attacks on your server.

q4q8H6MCeSo.jpg

The screen is taken from the repository with the program.

After installation, you can customize the output a little. For example, the following command will show no more than five entries from SSH logs:
Code:
/ usr / local / bin / log-stats ssh -c 5

All explanations are in the repository with the program.

asroute
If you are an engineer of some telecom operator, then you regularly have to tinker with traces, pings and large routers. For you, they wrote asroute - a utility that allows you to tracerouteadd information about the ASN of transit routers to the output . For example, it looks like this:
Code:
$ traceroute -a www.bhutan.gov.bt | asroute
traceroute to bhutan.gov.bt (202.144.128.217), 64 hops max, 52 byte packets
-> AS0 (Reserved)
-> *
-> BRESNAN-33588, US
-> LIGHTOWER, US
-> BRESNAN-33588, US
-> CHARTER-20115, US
-> TELIANET Telia Carrier, EU
-> *
-> NTT-COMMUNICATIONS-2914, US
-> DRUKNET-AS DrukNet ISP, BT
-> BTTELECOM-AS-AP Bhutan Telecom Ltd, BT

Installation is still available only for macOS, but it is done with one command.
Code:
$ brew install asroute

Asroute is written in Rust, so in theory it can be compiled for any other platform without problems, but at the time of this writing, I only have a Windows computer with me, which has known difficulties with compiling unconventional code.

Outrun
The highlight of the program is the outrun script, which allows you to easily move calculations outside the limits of a weak thin client to a powerful remote server, and the presence of the required utility on the remote host is not at all necessary. The only thing that is needed to work is the installed outrun on both hosts (both local and remote), and with the presence of root rights (chroot is used).

Demo from the authors of the program
As the most obvious example (even cited by the authors of outrun), you can run a local video converter like FFmpeg using server resources. The GIF shows a very significant difference in execution time.

You can install directly from pip.
Code:
pip3 install outrun

Then you can run the required command on the remote machine:
Code:
outrun srv4.local neofetch

The file system will be available as if it were local, and the results of the command (if any) will also be written to the current folder on the local computer.

WEB
Webify - broadcasting console output

With Webify, you can translate the output of almost any command as a web service. A good solution for simple console sharing: you launch Webify with your service, and the other person uncovers the browser (or curl) and interacts with your application.

tmpmail
This script is a great way to get a temporary mailbox right from the terminal. If you are reading "Hacker" - I obviously do not need to explain what it is and why you need it, so let's go straight to the installation.
Code:
$ sudo dnf install jq curl w3m
$ wget https://raw.githubusercontent.com/sdushantha/tmpmail/master/tmpmail
$ chmod + x tmpmail

We generate a box:
Code:
$ ./tmpmail --generate
qpasixkauet@1secmail.net

Checking mail:
Code:
./tmpmail --recent

ZeroSSL
You also thought that free SSL certificates can only be obtained from Let's Encrypt? Not anymore! The guys from ZeroSSL rolled out an alternative service where you can get all the same certificates for the same 90 days. You can use good old ACME. If for some reason I was looking for a replacement for the old Let's Encrypt - here it is.
f1cpLhPi4XE.jpg


The release procedure is completely straightforward: you just enter your mail and password and confirm ownership of the domain via DNS or by uploading the specified file to it. In a few minutes, the site will generate your certificate and offer to download it, and what to do next is not for me to teach you.

TestTLS
testtls.com is another service for SSL / TLS checking, an analogue of SSLLabs.
Ia9Gxop4ya0.jpg

Check result.

With its help, you can easily check if HTTPS is configured correctly on your (or someone else's) site.

Useful sites and services
Here are a few more links that are not in the command line utilities category but may be helpful.

Public API
Public API is a site that contains information about public interfaces of various services and links to documentation. It can be useful in a variety of cases - from marketing to OSINT.

Gitexplorer
GitExplorer is a good interactive Git cheat sheet.

1VYnfbz8xBI.jpg

Grep.app
Now we can, but grepnut around GitHub we require entry directly from the browser. It is useful for finding secrets during penetration testing (or on the bug bounty program) or flags from CTF, for example.

gWaf-05ia1g.jpg


Authenticator
If you use two-factor authentication, you would clearly be pleased to have the keys to enter at hand, and not in the phone, which still needs to be found. Authenticator plugin for Chrome, Firefox and Microsoft Edge can generate codes for 2FA right in the browser. Also makes an encrypted backup to Google Drive, Dropbox or OneDrive. Supports classic TOTP and HOTP, as well as Steam Guard and Blizzard Authenticator, in case you use these sites.
ayhR_iFJ5dM.jpg


Free for Dev
Free for Dev is a collection of free services that can be useful for developers. Of course, not all of them are free, but everywhere there is a period or limit of free use, which is almost always more than enough for personal use or tests.
R80jhybZFqU.jpg


RESULTS
Of course, you can say that it is easy to live without all this, and to automate routine tasks with self-written scripts. But why, if there are ready-made debugged tools that will improve your work for free and save time and nerves? A hacker is a hacker because he can make his workplace better than others.
BaVR5h_tlO4.jpg


To be continued…
 
Top