Blog Archive, Scanner / 17 3月 2015 / Essentia Team

Needle in a Haystack. Using Essentia to organize log files

It’s simple: big data means lots of logs and these logs tend to be disordered and very hard to distinguish. If the desired log files are in different directories and particularly if there are other log files in those directories along side them, it can sometimes be necessary to specify these files and their paths by name. This is an incredibly messy and time consuming process that Essentia remedies.

With the Essentia Scanner you simply point to your datastore–where your files are being stored–and the list of filenames are stored in a database file. You can then easily explore how these files are organized and categorize your files into the segments you want them to be in.

These categories are then easily streamed directly into whatever command you want (such as the Essentia Preprocessor) so you they can be analyzed or explored further. If your logs have a timestamp in their filenames, you can easily narrow down the streamed data to whatever time frame you want to send to your chosen command.

Essentia’s powerful scanning capabilities make it easy to organize your data and analyze the data you want in the time frame you want it to be analyzed.