## "Fossies" - the Fresh Open Source Software Archive

### Member "gretl-2020e/doc/tex/appendices.tex" (23 Feb 2020, 25061 Bytes) of package /linux/misc/gretl-2020e.tar.xz:

As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) TeX and LaTeX source code syntax highlighting (style: standard) with prefixed line numbers. Alternatively you can here view or download the uninterpreted source code file.

    1 \chapter{Data file details}
2 \label{app-datafile}
3
4 \section{Basic native format}
5 \label{native}
6
7 In gretl's basic native data format--for which we use the suffix
8 \texttt{gdt}---a data set is stored in XML (extensible mark-up
9 language). Data files correspond to the simple DTD (document type
10 definition) given in \verb+gretldata.dtd+, which is supplied with the
11 gretl distribution and is installed in the system data directory
12 (e.g.\ \url{/usr/share/gretl/data} on Linux.)  Such files may be plain
13 text (uncompressed) or gzipped.  They contain the actual data values
14 plus additional information such as the names and descriptions of
15 variables, the frequency of the data, and so on.
16
17 In a \texttt{gdt} file the actual data values are written to 17
18 significant figures (for generated data such as logs or pseudo-random
19 numbers) or to a maximum of 15 figures for primary data. The C
20 \texttt{printf} format \verb|%.*g|'' is used (for \texttt{*} = 17 or
21 15) so that trailing zeros are not printed.
22
23 Most users will probably not have need to read or write such files
24 other than via gretl itself, but if you want to manipulate them
25 using other software tools you should examine the DTD and also take a
26 look at a few of the supplied practice data files: \verb+data4-1.gdt+
27 gives a simple example; \verb+data4-10.gdt+ is an example where
28 observation labels are included.
29
30 \section{Binary data file format}
31 \label{bindata}
32
33 As of gretl 1.9.15, an alternative, binary format is available for
34 data storage. Files of this sort have suffix \texttt{gdtb}, and they
35 take the form of a \app{PKZIP} archive containing two files,
36 \texttt{data.xml} and \texttt{data.bin}, with the following
37 characteristics.
38 \begin{itemize}
39 \item \texttt{data.xml} is an XML file conforming to the gretldata DTD
40   mentioned above, holding all the metadata.
41 \item \texttt{data.bin} starts with a header composed of one of the
42   strings
43
44   \texttt{gretl-bin:little-endian} \,\,or \\
45   \texttt{gretl-bin:big-endian}
46
47   padded to 24 bytes with nul characters.  This is followed by a
48   binary dump of the data series, by variable, as double-precision
49   floating-point values.
50 \end{itemize}
51
52 Binary values are saved in the endianness of the machine on which
53 they're written; the header information enables gretl to convert to
54 the endianness of the host on which the data are read if need be.
55
56 The rationale for introducing the binary \texttt{gdtb} format is that
57 for very large datasets it is a lot faster to write and read data in
58 this form rather than as text. For small to moderately sized datasets
59 (say, up to 10 megabytes or so) there is no appreciable advantage in the
60 binary format and we recommend use of plain \texttt{gdt}.
61
62 Some illustrative timings are shown in Table~\ref{tab:dataspeed};
63 these were obtained on a Lenovo ThinkPad X1 Carbon running gretl
64 1.9.15.  The datasets contained a mixture of random normal series and
65 random binary (dummy) series. The largest comprised 50000 observations
66 on 1000 series and the smallest 5000 observations on 250 series.  As
67 can be seen, there is a big time saving from the binary format when
68 writing (and to a lesser extent, when reading) a dataset in the
69 hundreds of megabytes range. At a size of around 10 megabytes,
70 however, \texttt{gdt} files can be both written and read in under a
71 second, surely fast enough for most purposes.
72
73 \begin{table}[htbp]
74   \centering
75   \begin{tabular}{lrr@{\hskip 2em}rr@{\hskip 2em}rr@{\hskip 2em}rr}
76     & \multicolumn{2}{c}{381\,MB}
77     & \multicolumn{2}{c}{38\,MB}
78     & \multicolumn{2}{c}{19\,MB}
79     & \multicolumn{2}{c}{10\,MB} \\
81     \texttt{gdt} &
82     21.76 & 4.91 & 2.20 & 0.48 & 1.19 & 0.32 & 0.59 & 0.16\\
83     \texttt{gdtb} &
84     3.82 & 1.10 & 0.39 & 0.11 & 0.25 & 0.06 & 0.13 & 0.03
85   \end{tabular}
86   \caption{Data write and read timings in seconds for datasets
87     of various sizes in megabytes. The MB numbers represent the
88     size of the datasets in memory; files of both formats are
89     substantially smaller when compression is applied.}
90   \label{tab:dataspeed}
91 \end{table}
92
93
94 \section{Native database format}
95 \label{dbdetails}
96
97 A gretl database consists of two parts: an ASCII index file (with
98 filename suffix \texttt{.idx}) containing information on the series,
99 and a binary file (suffix \texttt{.bin}) containing the actual data.
100 Two examples of the format for an entry in the \texttt{idx} file are
101 shown below:
102
103 \begin{code}
104 G0M910  Composite index of 11 leading indicators (1987=100)
105 M 1948.01 - 1995.11  n = 575
106 currbal Balance of Payments: Balance on Current Account; SA
107 Q 1960.1 - 1999.4 n = 160
108 \end{code}
109
110 The first field is the series name.  The second is a description of
111 the series (maximum 128 characters).  On the second line the first
112 field is a frequency code: \texttt{M} for monthly, \texttt{Q} for
113 quarterly, \texttt{A} for annual, \texttt{B} for business-daily (daily
114 with five days per week), \texttt{D} for 7-day daily, \texttt{S} for
115 6-day daily, \texttt{U} for undated.
116
117 No other frequencies are accepted at present.  Then comes the starting
118 date (N.B. with two digits following the point for monthly data, one
119 for quarterly data, none for annual), a space, a hyphen, another
120 space, the ending date, the string \verb+n = +'' and the integer
121 number of observations. In the case of daily data the starting and
122 ending dates should be given in the form \verb+YYYY-MM-DD+. This
123 format must be respected exactly.
124
125 Optionally, the first line of the index file may contain a short
126 comment (up to 64 characters) on the source and nature of the data,
127 following a hash mark.  For example:
128
129 \begin{code}
130 # Federal Reserve Board (interest rates)
131 \end{code}
132
133 The corresponding binary database file holds the data values,
134 represented as floats'', that is, single-precision floating-point
135 numbers, typically taking four bytes apiece.  The numbers are packed
136 by variable'', so that the first \emph{n} numbers are the
137 observations of variable 1, the next \emph{m} the observations on
138 variable 2, and so on.
139
140 \chapter{Building gretl}
141 \label{app-build}
142
143 Here we give instructions detailed enough to allow a user
144 with only a basic knowledge of a Unix-type system to build gretl.
145 These steps were tested on a fresh installation of Debian Etch. For
146 other Linux distributions (especially Debian-based ones, like Ubuntu
147 and its derivatives) little should change. Other Unix-like operating
148 systems such as Mac OS X and BSD would probably require more substantial
150
151 In this guided example, we will build gretl complete with
152 documentation.  This introduces a few more requirements, but gives you
153 the ability to modify the documentation files as well, like the help
154 files or the manuals.
155
156 \section{Installing the prerequisites}
157
158 We assume that the basic GNU utilities are already installed on the
159 system, together with these other programs:
160 \begin{itemize}
161 \item some \TeX/\LaTeX system (\texttt{texlive} will do beautifully)
162 \item Gnuplot
163 \item ImageMagick
164 \end{itemize}
165 We also assume that the user has administrative privileges and knows
166 how to install packages.  The examples below are carried out using the
167 \texttt{apt-get} shell command, but they can be performed with
168 menu-based utilities like \texttt{aptitude}, \texttt{dselect} or the
169 GUI-based program \texttt{synaptic}. Users of Linux distributions
170 which employ rpm packages (e.g.\ Red Hat/Fedora, Mandriva, SuSE) may
171 want to refer to the
172 \href{http://gretl.sourceforge.net/depend.html}{dependencies} page on
173 the gretl website.
174
175 The first step is installing the C compiler and related basic
176 utilities, if these are not already in place. On a Debian (or
177 derivative) system, these are contained in a bunch of packages that
178 can be installed via the command
179 \begin{code}
180 apt-get install gcc autoconf automake1.9 libtool flex bison gcc-doc \
181 libc6-dev libc-dev gfortran gettext pkgconfig
182 \end{code}
183
184 Then it is necessary to install the development'' (\texttt{dev})
185 packages for the libraries that gretl uses:
186 \begin{center}
187   \begin{tabular}{ll}
188     \textit{Library} & \textit{command} \\ [4pt]
189     GLIB     & \texttt{apt-get install libglib2.0-dev} \\
190     GTK 3.0  & \texttt{apt-get install libgtk3.0-dev} \\
191     PNG      & \texttt{apt-get install libpng12-dev} \\
192     XSLT     & \texttt{apt-get install libxslt1-dev} \\
193     LAPACK   & \texttt{apt-get install liblapack-dev} \\
194     FFTW     & \texttt{apt-get install libfftw3-dev} \\
196     ZLIB     & \texttt{apt-get install zlib1g-dev} \\
197     XML      & \texttt{apt-get install libxml2-dev} \\
198     GMP      & \texttt{apt-get install libgmp-dev} \\
199     CURL     & \texttt{apt-get install libcurl4-gnutls-dev} \\
200     MPFR     & \texttt{apt-get install libmpfr-dev}
201   \end{tabular}
202 \end{center}
203
204 It is possible to substitute GTK 2.0 for GTK 3.0.  The \texttt{dev}
205 packages for these libraries are necessary to \emph{compile}
206 gretl---you'll also need the plain, non-\texttt{dev} library packages
207 to \emph{run} gretl, but most of these should already be part of a
208 standard installation.  In order to enable other optional features,
209 like audio support, you may need to install more libraries.
210
211 \tip{The above steps can be much simplified on Linux systems
212 that provide deb-based package managers, such as Debian and its
213 derivatives (Ubuntu, Knoppix and other distributions). The command
214
215 \texttt{apt-get build-dep gretl}
216
217 will download and install all the necessary packages for building the
218 version of gretl that is currently present in your APT
219 sources. Techincally, this does not guarantee that all the software
220 necessary to build the git version is included, because the version of
221 gretl on your repository may be quite old and build requirements
222 may have changed in the meantime. However, the chances of a mismatch
223 are rather remote for a reasonably up-to-date system, so in most cases
224 the above command should take care of everything correctly.}
225
226 \section{Getting the source: release or git}
227
228 At this point, it is possible to build from the source.  You have two
229 options here: obtain the latest released source package, or retrieve
230 the current git version of gretl (git = the version control software
231 currently in use for gretl).  The usual caveat applies to the git
232 version, namely, that it may not build correctly and may contain
233 experimental'' code; on the other hand, git often contains bug-fixes
234 relative to the released version.  If you want to help with testing
235 and to contribute bug reports, we recommend using git gretl.
236
237 To work with the released source:
238 \begin{enumerate}
240   \href{http://gretl.sourceforge.net/}{gretl.sourceforge.net}.
241 \item Unzip and untar the package.  On a system with the GNU utilities
242   available, the command would be \cmd{tar xvfJ gretl-N.tar.xz}
243   (replace \cmd{N} with the specific version number of the file you
245 \item Change directory to the gretl source directory created at step 2
246   (e.g.\ \verb+gretl-1.10.2+).
247 \item Proceed to the next section, Configure and make''.
248 \end{enumerate}
249
250 To work with git you'll first need to install the \app{git} client
251 program if it's not already on your system.  Relevant resources you
252 may wish to consult include the main git website at
253 \href{https://git-scm.com/}{git-scm.com} and instructions specific to
254 gretl:
255 \href{http://gretl.sourceforge.net/gretl-git-basics.html}{gretl
256   git basics}.
257
258 When grabbing the git sources \textit{for the first time}, you should
259 first decide where you want to store the code.  For example, you might
260 create a directory called \texttt{git} under your home directory.
261 Open a terminal window, \texttt{cd} into this directory, and type
262 the following commands:
263 %
264 \begin{code}
265 git clone git://git.code.sf.net/p/gretl/git gretl-git
266 \end{code}
267 %
268 At this point \app{git} should create a subdirectory named
269 \texttt{gretl-git} and fill it with the current sources.
270
271 When you want to \textit{update the source}, this is very simple: just
272 move into the \texttt{gretl-git} directory and type
273 \begin{code}
274 git pull
275 \end{code}
276
277 Assuming you're now in the \texttt{gretl-git} directory, you can
278 proceed in the same manner as with the released source package.
279
280
281 \section{Configure the source}
282
283 The next command you need is \texttt{./configure}; this is a complex
284 script that detects which tools you have on your system and sets
285 things up. The \texttt{configure} command accepts many
286 options; you may want to run
287 \begin{code}
288 ./configure --help
289 \end{code}
290 first to see what options are available. One option you way wish to
291 tweak is \cmd{--prefix}.  By default the installation goes under
292 \verb+/usr/local+ but you can change this.  For example
293 \begin{code}
294 ./configure --prefix=/usr
295 \end{code}
296 will put everything under the \verb+/usr+ tree.
297
298 If you have a multi-core machine you may want to activate support
299 for OpenMP, which permits the parallelization of matrix
301 \texttt{configure} flag
302 \begin{code}
303 --enable-openmp
304 \end{code}
305
306 By default the gretl GUI is built using version 3.0 of the GTK
307 library, if available, otherwise version 2.0. If you have both
308 versions installed and prefer to use GTK 2.0, use the flag
309 \begin{code}
310 --enable-gtk2
311 \end{code}
312
313 In order to have the documentation built, we need to pass the relevant
314 option to \texttt{configure}, as in
315 \begin{code}
316 --enable-build-doc
317 \end{code}
318 But please note that this option will work only if you are using
319 the git source.
320
321
322 \tip{In order to build the documentation, there is the possibility
323   that you will have to install some extra software on top of the
324   packages mentioned in the previous section. For example, you may
325   need some extra \LaTeX\ packages to compile the manuals. Two of the
326   required packages, that not every standard \LaTeX\ installation
327   include, are typically \app{pifont.sty} and \app{appendix.sty}. You
328   could install the corresponding packages from your distribution or
329   you could simply download them from CTAN and install them by hand.}
330
331
332 This, for example, if you want to install under \texttt{/usr}, with
333 OpenMP support, and also build the documentation, you would do
334 \begin{code}
335 ./configure --prefix=/usr \
336  --enable-openmp \
337  --enable-build-doc
338 \end{code}
339
340 You will see a number of checks being run, and if everything goes
341 according to plan, you should see a summary similar to that displayed
342 in Listing~\ref{configure-output}.
343
344 \begin{script}[htbp]
345   \caption{Sample output from \texttt{./configure}}
346   \label{configure-output}
347 \begin{scode}
348 Configuration:
349
350   Installation path:                      /usr
352   Use gnuplot for graphs:                 yes
353   Use LaTeX for typesetting output:       yes
354   Use libgsf for zip/unzip:               no
355   sse2 support for RNG:                   yes
356   OpenMP support:                         yes
357   MPI support:                            no
358   AVX support for arithmetic:             no
359   Build with GTK version:                 2.0
360   Build gretl documentation:              yes
361   Use Lucida fonts:                       no
362   Build message catalogs:                 yes
363   X-12-ARIMA support:                     yes
364   TRAMO/SEATS support:                    yes
365   libR support:                           yes
366   ODBC support:                           no
367   Experimental audio support:             no
368   Use xdg-utils in installation:          if DESTDIR not set
369   LAPACK libraries:
370     -llapack -lblas -lgfortran
371
372 Now type 'make' to build gretl.
373 You can also do 'make pdfdocs' to build the PDF documentation.
374 \end{scode}
375 \end{script}
376
377 \tip{If you're using git, it's a good idea to re-run the
378   \texttt{configure} script after doing an update.  This is not always
379   necessary, but sometimes it is, and it never does any harm.  For
380   this purpose, you may want to write a little shell script that calls
381   \texttt{configure} with any options you want to use.}
382
383
384 \section{Build and install}
385
386 We are now ready to undertake the compilation proper: this is done by
387 running the \texttt{make} command, which takes care of compiling all
388 the necessary source files in the correct order. All you need to do is
389 type
390 \begin{code}
391 make
392 \end{code}
393
394 This step will likely take several minutes to complete; a lot of
395 output will be produced on screen. Once this is done, you can install
397 \begin{code}
398 make install
399 \end{code}
400
401 On most systems, the \texttt{make install} command requires you to
403 \texttt{root} before launching \texttt{make install} or you may want
404 to use the \texttt{sudo} utility, as in:
405 \begin{code}
406 sudo make install
407 \end{code}
408
409 Now try if everything works: go back to your home directory and run gretl
410 \begin{code}
411 cd ~
412 gretl &
413 \end{code}
414
415 If all is well, you ought to see gretl start, at which point just exit
416 the program in the usual way. On the other hand, there is the
417 possibility that gretl doesn't start and instead you see a message
418 like
419
420 \begin{code}
422    libgretl-1.0.so.0: cannot open shared object file: No such file or directory
423 \end{code}
424
425 In this case, just run
426 \begin{code}
427   sudo ldconfig
428 \end{code}
429 The problem should be fixed once and for all.
430
431 \chapter{Numerical accuracy}
432 \label{app-accuracy}
433
434 Gretl uses double-precision arithmetic throughout---except for
435 the multiple-precision plugin invoked by the menu item Model, Other
436 linear models, High precision OLS'' which represents floating-point
437 values using a number of bits given by the environment variable
438 \verb+GRETL_MP_BITS+ (default value 256).
439
440 The normal equations of Least Squares are by default solved via
441 Cholesky decomposition, which is highly accurate provided the matrix
442 of cross-products of the regressors, $X'X$, is not very ill
443 conditioned.  If this problem is detected, gretl automatically
444 switches to use QR decomposition.
445
446 The program has been tested rather thoroughly on the statistical
447 reference datasets provided by NIST (the U.S.  National Institute of
448 Standards and Technology) and a full account of the results may be
449 found on the gretl website (follow the link Numerical accuracy'').
450
451 To date, two published reviews have discussed gretl's accuracy:
452 Giovanni Baiocchi and Walter Distaso \citeyearpar{baiocchi03}, and
453 Talha Yalta and Yasemin Yalta \citeyearpar{yalta07}.  We are grateful
454 to these authors for their careful examination of the program.  Their
455 comments have prompted several modifications including the use of
456 Stephen Moshier's \app{cephes} code for computing p-values and other
457 quantities relating to probability distributions (see
458 \href{http://www.netlib.org/cephes/}{netlib.org}), changes to the
459 formatting of regression output to ensure that the program displays a
460 consistent number of significant digits, and attention to compiler
461 issues in producing the MS Windows version of gretl (which at
462 one time was slighly less accurate than the Linux version).
463
464 Gretl now includes a plugin'' that runs the NIST linear
465 regression test suite.  You can find this under the Tools'' menu in
466 the main window.  When you run this test, the introductory text
467 explains the expected result.  If you run this test and see anything
468 other than the expected result, please send a bug report to
469 \verb+cottrell@wfu.edu+.
470
471 All regression statistics are printed to 6 significant figures in the
472 current version of gretl (except when the multiple-precision
473 plugin is used, in which case results are given to 12 figures).  If
474 you want to examine a particular value more closely, first save it
475 (for example, using the \cmd{genr} command) then print it using
476 \cmd{printf}, to as many digits as you like (see the \GCR).
477
478 \chapter{Related free software}
480
481 Gretl's capabilities are substantial, and are expanding.
482 Nonetheless you may find there are some things you can't do in
483 gretl, or you may wish to compare results with other programs.
484 If you are looking for complementary functionality in the realm of
485 free, open-source software we recommend the following programs.  The
486 self-description of each program is taken from its website.
487
488 \begin{itemize}
489
490 \item \textbf{GNU R} \href{http://www.r-project.org/}{r-project.org}:
491   R is a system for statistical computation and graphics. It
492   consists of a language plus a run-time environment with graphics, a
493   debugger, access to certain system functions, and the ability to run
494   programs stored in script files\dots\ It compiles and runs on a wide
495   variety of UNIX platforms, Windows and MacOS.''  Comment: There are
496   numerous add-on packages for R covering most areas of statistical
497   work.
498
499 \item \textbf{GNU Octave}
500   \href{http://www.octave.org/}{www.octave.org}:
501   GNU Octave is a high-level language, primarily intended for
502   numerical computations. It provides a convenient command line
503   interface for solving linear and nonlinear problems numerically, and
504   for performing other numerical experiments using a language that is
505   mostly compatible with Matlab. It may also be used as a
506   batch-oriented language.''
507
508 \item \textbf{Julia} \href{http://julialang.org/}{julialang.org}:
509   Julia is a high-level, high-performance dynamic programming
510   language for technical computing, with syntax that is familiar to
511   users of other technical computing environments. It provides a
512   sophisticated compiler, distributed parallel execution, numerical
513   accuracy, and an extensive mathematical function library.''
514
515 \item \textbf{JMulTi} \href{http://www.jmulti.de/}{www.jmulti.de}:
516   JMulTi was originally designed as a tool for certain econometric
517   procedures in time series analysis that are especially difficult to
518   use and that are not available in other packages, like Impulse
519   Response Analysis with bootstrapped confidence intervals for VAR/VEC
520   modelling. Now many other features have been integrated as well to
521   make it possible to convey a comprehensive analysis.''  Comment:
522   JMulTi is a java GUI program: you need a java run-time environment to
523   make use of it.
524
525 \end{itemize}
526
527 As mentioned above, gretl offers the facility of exporting
528 data in the formats of both Octave and R.  In the case of Octave, the
529 gretl data set is saved as a single matrix, \verb+X+. You can
530 pull the \verb+X+ matrix apart if you wish, once the data are loaded
531 in Octave; see the Octave manual for details.  As for R, the exported
532 data file preserves any time series structure that is apparent to
533 gretl.  The series are saved as individual structures. The data
534 should be brought into R using the \cmd{source()} command.
535
536 In addition, gretl has a convenience function for moving data
537 quickly into R.  Under gretl's Tools'' menu, you will find the
538 entry Start GNU R''.  This writes out an R version of the current
539 gretl data set (in the user's gretl directory), and sources it
540 into a new R session.  The particular way R is invoked depends on the
541 internal gretl variable \verb+Rcommand+, whose value may be set
542 under the Tools, Preferences'' menu.  The default command is
543 \cmd{RGui.exe} under MS Windows. Under X it is \cmd{xterm -e R}.
544 Please note that at most three space-separated elements in this
545 command string will be processed; any extra elements are ignored.
546
547 \chapter{Listing of URLs}
548 \label{app-urls}
549
550 Below is a listing of the full URLs of websites mentioned in the text.
551
552 \begin{description}
553
554 \item[Estima (RATS)] \url{http://www.estima.com/}
555 \item[FFTW3] \url{http://www.fftw.org/}
556 \item[Gnome desktop homepage] \url{http://www.gnome.org/}
557 \item[GNU Multiple Precision (GMP) library]
558   \url{http://gmplib.org/}
559 \item[CURL library]
560   \url{http://curl.haxx.se/libcurl/}
561 \item[GNU Octave homepage] \url{http://www.octave.org/}
562 \item[GNU R homepage] \url{http://www.r-project.org/}
563 \item[GNU R manual]
564   \url{http://cran.r-project.org/doc/manuals/R-intro.pdf}
565 \item[Gnuplot homepage] \url{http://www.gnuplot.info/}
566 \item[Gnuplot manual] \url{http://ricardo.ecn.wfu.edu/gnuplot.html}
567 \item[Gretl data page]
568   \url{http://gretl.sourceforge.net/gretl_data.html}
569 \item[Gretl homepage] \url{http://gretl.sourceforge.net/}
570 \item[GTK+ homepage] \url{http://www.gtk.org/}
571 \item[GTK+ port for win32]
572   \url{https://wiki.gnome.org/Projects/GTK/Win32/}
573 \item[InfoZip homepage]
574   \url{http://www.info-zip.org/pub/infozip/zlib/}
575 \item[JMulTi homepage] \url{http://www.jmulti.de/}
576 \item[JRSoftware] \url{http://www.jrsoftware.org/}
577 \item[Julia homepage] \url{http://julialang.org/}
578 \item[Mingw (gcc for win32) homepage] \url{http://www.mingw.org/}
579 \item[Minpack] \url{http://www.netlib.org/minpack/}
580 \item[Penn World Table] \url{http://pwt.econ.upenn.edu/}