% vim:tw=110:tabstop=4:shiftwidth=4 %\documentclass[11pt]{report} \documentclass[10pt,twocolumn]{article} % Specify what packages we're using \usepackage{times} \usepackage{textcomp} \usepackage{lastpage} \usepackage{latexsym} \usepackage{amssymb} \usepackage{listings} \usepackage[bottom, multiple, marginal, norule]{footmisc} \usepackage[dvips]{graphicx} \usepackage[colorlinks, bookmarks, dvips]{hyperref} \usepackage{fancyvrb} \usepackage[footnotesize,bf]{caption2} \usepackage{rotating} % Use the geometry package and fiddle with margins % Don't mess with this :-P \usepackage[letterpaper,lmargin=.65in,rmargin=.65in,bottom=.5in,top=.5in,headsep=.6in]{geometry} % with 7 mm gutter \setlength{\columnsep}{.7cm} % Separate footnotes (if present) from the margins by a reasonable amount \setlength{\captionmargin}{.5in} % Set the footnote indent \setlength{\footnotemargin}{0.25em} % Manually set the footnote separation from the body text by redefining what footnoterule means % (Note that this overrides the 'norule' in the footmisc options) %\renewcommand\footnoterule{% % \vskip 0.25in % \hrule\@width.4\columnwidth} % Set our standard unit of measure \setlength{\unitlength}{1in} % Change the behaviour of floats to something a little more liberal \renewcommand\floatpagefraction{.9} \renewcommand\topfraction{.9} \renewcommand\bottomfraction{.9} \renewcommand\textfraction{.01} \setcounter{totalnumber}{50} \setcounter{topnumber}{50} \setcounter{bottomnumber}{50} % Create our custom config-file printing environment % Use this for anything you want to print out "code style" % like so: \begin{Verbatim} blah \end{Verbatim} % Note: {Verbatim} is case sensitive! Capital V, please. % (this is based on the fancyvrb package) \RecustomVerbatimEnvironment{Verbatim}{Verbatim}{fontsize=\scriptsize,frame=lines,obeytabs=true,tabsize=4} \fvset{fontsize=\scriptsize,frame=lines,obeytabs=trie,tabsize=4} % For "little" code snippets (like URLs or email address), refine the + % symbol to act as a formatting delimiter,.i.e. +code+ would be formatted % in a way to mark it as a code snippet. \DefineShortVerb{\+} % Specify the document's meta information \title{Automated Report Generaton\\ With \LaTeX{} \& MetaPost} \author{\textsl{Tillman Hodgson}} \date{Initial Revision: March 4, 2003\\ Revision date: \today} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % This is where the content begins ... \begin{document} % Create our standard title out of our document meta information \maketitle % Create and format the table of contents %{ % \setlength{\parskip}{0pt} % \tableofcontents %} % Define how we want our paragraphs to look % Note: This is done after the TOC so that it doesn't affect how % the TOC looks \setlength{\parindent}{0pt} \setlength{\parskip}{1.5ex plus 0.2ex minus 0.1ex} % Set up our page style for future pages (beyond the TOC) \pagestyle{headings} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Background} \subsection{The motivation} We've all been there. Slaving away creating Yet Another Monthly Report\footnote{YAMR, as in ``I was yammer'ing at the manager the other day \ldots''}. The particular process I has was particularly\footnote{Or typically, depending on your experiences} torturous: \begin{itemize} \item Using a Java based tool to generate a report of raw numbers that are displayed in a web page \item Cut and paste the values into spreadsheet in order to generate a chart \item Cut and paste the chart into a word processor in order to generate a report \item Try and remember which group of people is supposed to receive the report and email it to them \end{itemize} This was getting ridiculous --- I'm a Unix guru, after all! \subsection{The inspiration} I admit it, I love \LaTeX. It's a manly tool with the finesse of a fencing blade and a depth that's astounding. It strikes me as being fundamentally {\it correct} in how it's put together, and it's the closest I've ever felt to having a printing press right at my fingertips\footnote{Warning: don't put your fingers too close to a printing press, they're called {\it presses} for a reason}. It has a daunting learning cliff and doesn't forgive mistakes. In other words, it's a perfect tool for Unix users. \LaTeX{} is fundamentally a Turing-complete programming language wrapped around the idea of desktop publishing\footnote{In the traditional sense of typesetting, not the new-fangled Publishing Wizurd sense}. It's a very flexible tool and can be easily automated at the Unix command line. Assuming that I could find a way to: \begin{enumerate} \item Generate performance stats \item Generates graphs based on the stats \item Wrap a report around the graphs \end{enumerate} I'd be able to spend my month-end whittlin' away at a script like a happy hacker rather than digging the cut and paste trench. The challenge was on! The performance stats were easy --- these are well-understood\footnote{Warning: the author is pulling wool over your eyes} tools of the trade. That left only the graph generation. Within short order I'd discovered that MetaPost would generate nice looking graphs\footnote{I've also used Perl::GD and gnuplot but now prefer MetaPost} and integrated nicely with \LaTeX. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Tools of the trade} \subsection{The usual suspects} On a BSD based Unix, the usual suspects for non-interactive tools would be {\tt vmstat}, {\tt iostat}, {\tt uptime}\footnote{To obtain the load average}, {\tt ps} and friends. There are also some interactive looks like {\tt systat} and {\tt top} and a whole heap of specialized tools. These aren't interesting to us precisely because they're interactive, and thus more difficult to script. The right tool for the job and all that. By the way, FreeBSD contains a great man page that covers the basics of tuning ({\tt tuning(7)})\footnote{Available on the web at {\tt\scriptsize http://www.freebsd.org/cgi/man.cgi?query=tuning}}. If you're new to tuning, it's definitely worth a read. \subsection{But is there something better?} I'm a fan of {\tt bsdsar}\footnote{You can find it the FreeBSD ports collection, naturally} which is a {\tt sar}-workalike. It gathers performance stats using the basic monitoring tools that any host should have, and then stores the result into a plain text database. You can then extract the details you're interested in later on. This is important because scripting the collection of performance stats is more complciated to get right than it seems. \subsection{Our contrived example} Our fictional environment is a single server. For keep our example simple, we'll also assume that you're only interested in CPU and disk space utilization. Since it handles the data store aspect automatically (and thus saves us a step), {\tt bsdsar} will be used in our contrived example. Naturally, this approach is merely a contrived example. If you actually consider this to be adequate performance monitoring I fondly suspect that you have yet to discover uses for swap space, networking, or databases\footnote{Among many other potential performance bottlenecks}. The reader is encouraged to implement something that might actually be useful for them. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Making the performance monitoring harness} \subsection{Define the goal} As per our contrived example, we want to obtain CPU and disk utilization. Hourly ``snapshots'' should be fine since performance monitoring focuses on general trends. \subsection{Get the data} {\tt bsdsar -d} gives us our disk operations per second and {\tt bsdsar -u} gives us all sorts of CPU stats. This should be all we need for raw data. Unfortunately, that was the easy part. Fortunately, the hard part is fun. \subsection{Massage the CPU data} We'll start with the CPU data since that seems to be a bit more complicated and we're manly Unix geeks. Or something like that. In any case, here's what the raw output of {\tt bsdsar -u} looks like: \begin{Verbatim} Time % User % Sys % Nice % Intrpt % Idle 00:00 30 6 0 1 63 01:00 2 1 0 1 96 02:00 27 11 0 1 61 03:00 9 2 0 0 90 04:00 1 0 0 1 98 05:00 16 2 0 0 82 06:00 13 3 0 2 83 07:00 25 5 0 2 68 08:00 42 14 0 0 44 08:20 9 4 0 1 86 08:40 24 9 0 0 66 09:00 46 13 0 2 40 09:20 27 9 0 1 63 09:40 23 12 0 2 64 10:00 8 5 0 0 87 \end{Verbatim} Hmmm. That only goes until 10:00 ... this must be {\it todays} data. Since we'll end up running this from {\tt cron}, we should use yesterdays data and get a full 24 hours worth. {\tt bsdsar} has an {\tt -n} option that takes a date in MMDD format. Some trickery with the {\tt date} command should be able to automate this: \begin{Verbatim} bsdsar -n `date -j -v-1d "+%m%d"` -u \end{Verbatim} Another problem is that we only want the hourly data and {\tt bsdsar} is giving us data for 20 minutes after and 40 minutes after the hour for the hours between 8:00 and 17:00. A quick bit of {\tt grep} work ought to get rid of the extra lines: \begin{Verbatim} grep -v ^[0-9][0-9]:[24]0 \end{Verbatim} While we've got the {\tt grep} tool out, we may as well filter out the column headings: \begin{Verbatim} grep ^[0-9] \end{Verbatim} MetaPost doesn't understand time values very well, unfortunately, so we'll have to convert those hours to simple integer format by dropping the redundant ``:00'' information: \begin{Verbatim} sed -e 's/:00//' \end{Verbatim} One last thing and I think we've got it. We're only interested in how busy the CPU is, which really means the inverse of how idle the CPU is. So the only column of data that we need is the {\tt \% idle} column. Besides, managers are happier when higher values mean that things are in better shape and this matches that interpretation better. \begin{Verbatim} awk '{print $1, " ", $6}' \end{Verbatim} Oops! We need to capture this in a file. Let's add this to the end: \begin{Verbatim} > cpu.data \end{Verbatim} So when we put it all together it looks something like this: \begin{Verbatim} bsdsar -n `date -j -v-1d "+%m%d"` -u | \ grep -v ^[0-9][0-9]:[24]0 | grep ^[0-9] | \ sed -e 's/:00//' | \ awk '{print $1, " ", $6}' > cpu.data \end{Verbatim} The output file {\tt cpu.data} looks like this: \begin{Verbatim} 00 73 01 77 02 77 03 72 04 97 05 78 06 84 07 67 08 71 09 37 10 87 11 77 12 34 13 91 14 63 15 86 16 89 17 86 18 69 19 81 20 71 21 78 22 80 23 64 \end{Verbatim} Perfect! \subsection{Massage the disk data} Following a similar process for disk utilization, we should be able to obtain the disk operations per second with something like: \begin{Verbatim} bsdsar -n `date -j -v-1d "+%m%d"` -d | \ grep -v ^[0-9][0-9]:[24]0 | grep ^[0-9] | \ sed -e 's/:00//' | \ awk '{print $1, " ", $2}' > disk.data \end{Verbatim} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Making the MetaPost skeleton} So now we have our performance monitoring data in some nice and simple plain text two-column files. This is handy because MetaPost can use these files directly with it's {\tt gdraw} command. To make a graph, MetaPost needs to be told to use it's graphing module. You then need to provide the details of the graph (such as its size and what to draw) inside of a ``fig'' statement. Lastly, you need to say {\tt bye} so that MetaPost knows when to stop processing the file. Here's what our initial skeleton to graph our CPU utilization graph looks like: \begin{Verbatim} input graph; beginfig(1); draw begingraph(3.5in,2.5in); gdraw "cpu.data"; endgraph; endfig; bye; \end{Verbatim} We're going to need to add some labels to the graph or it'll be hard to understand. We can use the {\tt glabel} command to do that. Note that you need to wrap your text in {\tt btex} and {\tt etex} statements so that it knows where your text begins and ends: \begin{Verbatim} input graph; beginfig(1); draw begingraph(3.5in,2.5in); glabel.bot(btex Hour etex, OUT); glabel.lft(btex CPU Percent Idle etex, OUT); gdraw "cpu.data"; endgraph; endfig; bye; \end{Verbatim} Lastly, we need to create our second graph for the disk utilization. Rotating the left-hand label would also look more professional: \begin{Verbatim} input graph; beginfig(1); draw begingraph(3.5in,2.5in); glabel.bot(btex Hour etex, OUT); glabel.lft(btex CPU \% Idle etex rotated 90, OUT); gdraw "cpu.data"; endgraph; endfig; beginfig(2); draw begingraph(3.5in,2.5in); glabel.bot(btex Hour etex, OUT); glabel.lft(btex Disk Ops/sec etex rotated 90, OUT); gdraw "disk.data"; endgraph; endfig; bye; \end{Verbatim} Save this final version to a file called {\tt perfgraphs.mp}. If you were to run the command {\tt mpost perfgraphs}\footnote{The {\tt .mp} file extension is optional with the {\tt mpost} command}, MetaPost would create two files ({\tt perfgraphs.1} and {\tt perfgraphs.2}). These files are your graphs, in encapsulated PostScript (EPS) format. They look like this: \vskip 0.25in \begin{figure}[h] \centering \includegraphics{perfgraphs.1} \end{figure} \begin{figure}[h] \centering \includegraphics{perfgraphs.2} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Making the \LaTeX{} skeleton} This isn't a \LaTeX{} tutorial\footnote{I recommend \emph{The Not So Short Introduction to \LaTeX2e{}} for that, available from CTAN}, so a simple bare-bones LaTeX skeleton is given here with no explanations as to how or why it works. At the very least you should add your own wording - after all, the graphs are pretty but a report should contain some witty analysis as well. \VerbatimInput{sample_latex.ltx} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Automating the works} \subsection{Making the makefile} \LaTeX{} can be time-consuming to work with by hand because the document must be ``compiled'' before it can be vewed. Automating this with a {\tt make} can turn this into a single command. To do that, let's create a {\tt Makefile}\footnote{I assume that you named your \LaTeX{} file {\tt report.ltx}}: \begin{Verbatim} ########################################### # Makefile for automated report generator # ########################################### pdf: mpost perfgraphs latex report.ltx latex report.ltx dvips -Ppdf -G0 report.dvi ps2pdf report.ps clean: rm -f *.aux rm -f *.log rm -f *.dvi rm -f *.toc rm -f *.out rm -f *.lof rm -f *.lot rm -f *.ps rm -f *.pdf \end{Verbatim} To make your report (in PDF format) just type {\tt make pdf} and a file called {\tt report.pdf} will be created. To get rid of all the temporary files that it creates, type {\tt make clean}\footnote{It's not necessary to do that though, \LaTeX{} will overwrite them if necessary}. Note that we run {\tt latex} twice in the {\tt Makefile}. That's due to the particularities of \LaTeX{} and how it creates things like a table of contents. We also us regular {\tt latex} and then convert the resulting PostScript file to PDF rather than using {\tt pdflatex} directly because {\tt pdflatex} can't handle the file format of our graphs (ecapsulated PostScript). Normally this would result in bitmapped fonts beign used (which look {\emph horrible} in PDF format), but clever use of the {\tt times} PostScript font package in our {\tt report.ltx} file and some options given to {\tt dvips} in our {\tt Makefile} take care of that. \subsection{Enter the cron job} We want this to run each day, sometime after midnight, using yesterdays {\tt bsdsar} data. After the report has been generated, we'd like to email to our manager. Something like this added to {\tt /etc/crontab} ought to work: \begin{Verbatim} 22 6 * * * root /path/to/report_Makefile/make pdf && \ mail -s 'Yesterdays report' boss@example.com \ < /path/to/report/report.pdf \end{Verbatim} That's about it for our contrived example. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Where to next?} Aside from the obvious ``monitor more/better performance metrics'', there are lots of areas where this little piece of automation could be further refined: \begin{itemize} \item What if you have more than one server? Centralized performance monitoring is a big topic and your choice of software can drastically affect your ability to automate report generation. \item The graphs can be made much fancier. The documents \emph{Drawing Graphs with MetaPost} and \emph{Some Experiences in Running MetaFont and MetaPost} (both available on CTAN) should be enough to get you started. \item The one part that's not automated is the entering of your textual analysis --- building some kind of system to facilitate this would be neat. Perhaps a web front-end so the non-\LaTeX{} crowd can use it as a ``black box'' solution or extracting the information directly from a daily operations log\footnote{You are keeping one of these, right?} would work. \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \end{document}