"Fossies" - the Fresh Open Source Software Archive

Member "cfitsio-4.0.0/drvrnet.c" (20 May 2021, 125780 Bytes) of package /linux/misc/cfitsio-4.0.0.tar.gz:


As a special service "Fossies" has tried to format the requested source page into HTML format using (guessed) C and C++ source code syntax highlighting (style: standard) with prefixed line numbers and code folding option. Alternatively you can here view or download the uninterpreted source code file. For more information about "drvrnet.c" see the Fossies "Dox" file reference documentation and the last Fossies "Diffs" side-by-side code changes report: 3430_vs_3440.

    1 /*  This file, drvrhttp.c contains driver routines for http, ftp and root 
    2     files. */
    3 
    4 /* This file was written by Bruce O'Neel at the ISDC, Switzerland          */
    5 /*  The FITSIO software is maintained by William Pence at the High Energy  */
    6 /*  Astrophysic Science Archive Research Center (HEASARC) at the NASA      */
    7 /*  Goddard Space Flight Center.                                           */
    8 
    9 
   10 /* Notes on the drivers:
   11 
   12    The ftp driver uses passive mode exclusivly.  If your remote system can't 
   13    deal with passive mode then it'll fail.  Since Netscape Navigator uses 
   14    passive mode as well there shouldn't be too many ftp servers which have
   15    problems.
   16 
   17 
   18    The http driver works properly with 301 and 302 redirects.  For many more 
   19    gory details see http://www.w3c.org/Protocols/rfc2068/rfc2068.  The only
   20    catch to the 301/302 redirects is that they have to redirect to another 
   21    http:// url.  If not, things would have to change a lot in cfitsio and this
   22    was thought to be too difficult.
   23    
   24    Redirects look like
   25 
   26 
   27    <HTML><HEAD>
   28    <TITLE>301 Moved Permanently</TITLE>
   29    </HEAD><BODY>
   30    <H1>Moved Permanently</H1>
   31    The document has moved <A HREF="http://heasarc.gsfc.nasa.gov/FTP/software/ftools/release/other/image.fits.gz">here</A>.<P>
   32    </BODY></HTML>
   33 
   34    This redirect was from apache 1.2.5 but most of the other servers produce 
   35    something very similiar.  The parser for the redirects finds the first 
   36    anchor <A> tag in the body and goes there.  If that wasn't what was intended
   37    by the remote system then hopefully the error stack, which includes notes 
   38    about the redirect will help the user fix the problem.
   39 
   40   ****************************************************************
   41    Note added in 2017:  
   42    The redirect format shown above is actually preceded by 2 lines that look like
   43   
   44    HTTP/1.1 302 Found
   45    LOCATION: http://heasarc.gsfc.nasa.gov/FTP/software/ftools/release/other/image.fits.gz
   46 
   47    The CFITSIO parser now looks for the "Location:" string, not the html tag.
   48   ****************************************************************
   49 
   50 
   51    Root protocal doesn't have any real docs, so, the emperical docs are as 
   52    follows.  
   53 
   54    First, you must use a slightly modified rootd server.  The modifications 
   55    include implimentation of the stat command which returns the size of the 
   56    remote file.  Without that it's impossible for cfitsio to work properly
   57    since fitsfiles don't include any information about the size of the files 
   58    in the headers.  The rootd server closes the connections on any errors, 
   59    including reading beyond the end of the file or seeking beyond the end 
   60    of the file.  The rootd:// driver doesn't reopen a closed connection, if
   61    the connection is closed you're pretty much done.
   62 
   63    The messages are of the form
   64 
   65    <len><opcode><optional information>
   66 
   67    All binary information is transfered in network format, so use htonl and 
   68    ntohl to convert back and forth.
   69 
   70    <len> :== 4 byte length, in network format, the len doesn't include the
   71          length of <len>
   72    <opcode> :== one of the message opcodes below, 4 bytes, network format
   73    <optional info> :== depends on opcode
   74 
   75    The response is of the same form with the same opcode sent.  Success is
   76    indicated by <optional info> being 0.
   77 
   78    Root is a NFSish protocol where each read/write includes the byte
   79    offset to read or write to.  As a result, seeks will always succeed
   80    in the driver even if they would cause a fatal error when you try
   81    to read because you're beyond the end of the file.
   82 
   83    There is file locking on the host such that you need to possibly
   84    create /usr/tmp/rootdtab on the host system.  There is one file per
   85    socket connection, though the rootd daemon can support multiple
   86    files open at once.
   87 
   88    The messages are sent in the following order:
   89 
   90    ROOTD_USER - user name, <optional info> is the user name, trailing
   91    null is sent though it's not required it seems.  A ROOTD_AUTH
   92    message is returned with any sort of error meaning that the user
   93    name is wrong.
   94 
   95    ROOTD_PASS - password, ones complemented, stored in <optional info>. Once
   96    again the trailing null is sent.  Once again a ROOTD_AUTH message is 
   97    returned
   98 
   99    ROOTD_OPEN - <optional info> includes filename and one of
  100      {create|update|read} as the file mode.  ~ seems to be dealt with
  101      as the username's login directory.  A ROOTD_OPEN message is
  102      returned.
  103 
  104    Once the file is opened any of the following can be sent:
  105 
  106    ROOTD_STAT - file status and size
  107    returns a message where <optional info> is the file length in bytes
  108 
  109    ROOTD_FLUSH - flushes the file, not sure this has any real effect
  110    on the daemon since the daemon uses open/read/write/close rather
  111    than the buffered fopen/fread/fwrite/fclose.
  112 
  113    ROOTD_GET - on send <optional info> includes a text message of
  114    offset and length to get.  Return is a status message first with a
  115    status value, then, the raw bytes for the length that you
  116    requested.  It's an error to seek or read past the end of the file,
  117    and, the rootd daemon exits and won't respond anymore.  Ie, don't
  118    do this.
  119 
  120    ROOTD_PUT - on send <optional info> includes a text message of
  121    offset and length to put.  Then send the raw bytes you want to
  122    write.  Then recieve a status message
  123 
  124 
  125    When you are finished then you send the message:
  126 
  127    ROOTD_CLOSE - closes the file
  128 
  129    Once the file is closed then the socket is closed.
  130 
  131 
  132 Revision 1.56  2000/01/04 11:58:31  oneel
  133 Updates so that compressed network files are dealt with regardless of
  134 their file names and/or mime types.
  135 
  136 Revision 1.55  2000/01/04 10:52:40  oneel
  137 cfitsio 2.034
  138 
  139 Revision 1.51  1999/08/10 12:13:40  oneel
  140 Make the http code a bit less picky about the types of files it
  141 uncompresses.  Now it also uncompresses files which end in .Z or .gz.
  142 
  143 Revision 1.50  1999/08/04 12:38:46  oneel
  144 Don's 2.0.32 patch with dal 1.3
  145 
  146 Revision 1.39  1998/12/02 15:31:33  oneel
  147 Updates to drvrnet.c so that less compiler warnings would be
  148 generated.  Fixes the signal handling.
  149 
  150 Revision 1.38  1998/11/23 10:03:24  oneel
  151 Added in a useragent string, as suggested by:
  152 Tim Kimball   Data Systems Division   kimball@stsci.edu   410-338-4417
  153 Space Telescope Science Institute     http://www.stsci.edu/~kimball/
  154 3700 San Martin Drive                 http://archive.stsci.edu/
  155 Baltimore MD 21218 USA                http://faxafloi.stsci.edu:4547/
  156 
  157    
  158  */
  159 
  160 #ifdef HAVE_NET_SERVICES
  161 #include <string.h>
  162 
  163 #include <sys/types.h>
  164 #include <netinet/in.h>
  165 #include <netinet/tcp.h>
  166 #include <sys/socket.h>
  167 #include <arpa/inet.h>
  168 #include <netdb.h>
  169 #include <errno.h>
  170 #include <stdio.h>
  171 #include <string.h>
  172 #include <stdlib.h>
  173 #include <math.h>
  174 
  175 #ifdef CFITSIO_HAVE_CURL
  176 #include <curl/curl.h>
  177 #endif
  178 
  179 #if defined(unix) || defined(__unix__)  || defined(__unix) || defined(HAVE_UNISTD_H)
  180 #include <unistd.h>  
  181 #endif
  182 
  183 #include <signal.h>
  184 #include <setjmp.h>
  185 #include "fitsio2.h"
  186 
  187 static jmp_buf env; /* holds the jump buffer for setjmp/longjmp pairs */
  188 static void signal_handler(int sig);
  189 
  190 /* Network routine error codes */
  191 #define NET_OK 0
  192 #define NOT_INET_ADDRESS -1000
  193 #define UNKNOWN_INET_HOST -1001
  194 #define CONNECTION_ERROR -1002
  195 
  196 /* Network routine constants */
  197 #define NET_DEFAULT 0
  198 #define NET_OOB 1
  199 #define NET_PEEK 2
  200 
  201 /* local defines and variables */
  202 #define MAXLEN 1200
  203 #define SHORTLEN 100
  204 static char netoutfile[MAXLEN];
  205 
  206 
  207 #define ROOTD_USER  2000       /*user id follows */
  208 #define ROOTD_PASS  2001       /*passwd follows */
  209 #define ROOTD_AUTH  2002       /*authorization status (to client) */
  210 #define ROOTD_FSTAT 2003       /*filename follows */
  211 #define ROOTD_OPEN  2004       /*filename follows + mode */
  212 #define ROOTD_PUT   2005       /*offset, number of bytes and buffer */
  213 #define ROOTD_GET   2006       /*offset, number of bytes */
  214 #define ROOTD_FLUSH 2007       /*flush file */
  215 #define ROOTD_CLOSE 2008       /*close file */
  216 #define ROOTD_STAT  2009       /*return rootd statistics */
  217 #define ROOTD_ACK   2010       /*acknowledgement (all OK) */
  218 #define ROOTD_ERR   2011       /*error code and message follow */
  219 
  220 typedef struct    /* structure containing disk file structure */ 
  221 {
  222   int sock;
  223   LONGLONG currentpos;
  224 } rootdriver;
  225 
  226 typedef struct  /* simple mem struct for receiving files from curl */
  227 {
  228    char *memory;
  229    size_t size;
  230 } curlmembuf;
  231 
  232 static rootdriver handleTable[NMAXFILES];  /* allocate diskfile handle tables */
  233 
  234 /* static prototypes */
  235 
  236 static int NET_TcpConnect(char *hostname, int port);
  237 static int NET_SendRaw(int sock, const void *buf, int length, int opt);
  238 static int NET_RecvRaw(int sock, void *buffer, int length);
  239 static int NET_ParseUrl(const char *url, char *proto, char *host, int *port, 
  240          char *fn);
  241 static int CreateSocketAddress(struct sockaddr_in *sockaddrPtr,
  242                    char *host,int port);
  243 static int ftp_status(FILE *ftp, char *statusstr);
  244 static int http_open_network(char *url, FILE **httpfile, char *contentencoding,
  245               int *contentlength);
  246 static int https_open_network(char *filename, curlmembuf* buffer);
  247 static int ftp_open_network(char *url, FILE **ftpfile, FILE **command, 
  248                 int *sock);
  249 static int ftps_open_network(char *filename, curlmembuf* buffer);
  250 static int ftp_file_exist(char *url);
  251 static int root_send_buffer(int sock, int op, char *buffer, int buflen);
  252 static int root_recv_buffer(int sock, int *op, char *buffer,int buflen);
  253 static int root_openfile(char *filename, char *rwmode, int *sock);
  254 static int encode64(unsigned s_len, char *src, unsigned d_len, char *dst);
  255 static int ssl_get_with_curl(char *url, curlmembuf* buffer, 
  256                 char* username, char* password);
  257 static size_t curlToMemCallback(void *buffer, size_t size, size_t nmemb, void *userp);
  258 static int curlProgressCallback(void *clientp, double dltotal, double dlnow,
  259                            double ultotal, double ulnow);
  260 
  261 /***************************/
  262 /* Static variables */
  263 
  264 static int closehttpfile;
  265 static int closememfile;
  266 static int closefdiskfile;
  267 static int closediskfile;
  268 static int closefile;
  269 static int closeoutfile;
  270 static int closecommandfile;
  271 static int closeftpfile;
  272 static FILE *diskfile;
  273 static FILE *outfile;
  274 
  275 static int curl_verbose=0;
  276 static int show_fits_download_progress=0;
  277 static unsigned int net_timeout = 360; /* in seconds */
  278 
  279 /*--------------------------------------------------------------------------*/
  280 /* This creates a memory file handle with a copy of the URL in filename. The 
  281    file is uncompressed if necessary */
  282 
  283 int http_open(char *filename, int rwmode, int *handle)
  284 {
  285 
  286   FILE *httpfile;
  287   char contentencoding[SHORTLEN];
  288   char errorstr[MAXLEN];
  289   char recbuf[MAXLEN];
  290   long len;
  291   int contentlength;
  292   int status;
  293   char firstchar;
  294 
  295   closehttpfile = 0;
  296   closememfile = 0;
  297 
  298   /* don't do r/w files */
  299   if (rwmode != 0) {
  300     ffpmsg("Can't open http:// type file with READWRITE access");
  301     ffpmsg("  Specify an outfile for r/w access (http_open)");
  302     goto error;
  303   }
  304 
  305   /* do the signal handler bits */
  306   if (setjmp(env) != 0) {
  307     /* feels like the second time */
  308     /* this means something bad happened */
  309     ffpmsg("Timeout (http_open)");
  310     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
  311     ffpmsg(errorstr);
  312     ffpmsg("   (multiplied x10 for files requiring uncompression)");
  313     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
  314     goto error;
  315   }
  316 
  317   (void) signal(SIGALRM, signal_handler);
  318   
  319   /* Open the network connection */
  320 
  321   if (http_open_network(filename,&httpfile,contentencoding,
  322                    &contentlength)) {
  323       alarm(0);
  324       ffpmsg("Unable to open http file (http_open):");
  325       ffpmsg(filename);
  326       goto error;
  327   } 
  328 
  329   closehttpfile++;
  330 
  331   /* Create the memory file */
  332   if ((status =  mem_create(filename,handle))) {
  333     ffpmsg("Unable to create memory file (http_open)");
  334     goto error;
  335   }
  336 
  337   closememfile++;
  338 
  339   /* Now, what do we do with the file */
  340   /* Check to see what the first character is */
  341   firstchar = fgetc(httpfile);
  342   ungetc(firstchar,httpfile);
  343   if (!strcmp(contentencoding,"x-gzip") || 
  344       !strcmp(contentencoding,"x-compress") ||
  345       strstr(filename,".gz") || 
  346       strstr(filename,".Z") ||
  347       ('\037' == firstchar)) {
  348     /* do the compress dance, which is the same as the gzip dance */
  349     /* Using the cfitsio routine */
  350 
  351     status = 0;
  352     /* Ok, this is a tough case, let's be arbritary and say 10*net_timeout,
  353        Given the choices for nettimeout above they'll probaby ^C before, but
  354        it's always worth a shot*/
  355     
  356     alarm(net_timeout*10);
  357     status = mem_uncompress2mem(filename, httpfile, *handle);
  358     alarm(0);
  359     if (status) {
  360       ffpmsg("Error writing compressed memory file (http_open)");
  361       ffpmsg(filename);
  362       goto error;
  363     }
  364     
  365   } else {
  366     /* It's not compressed, bad choice, but we'll copy it anyway */
  367     if (contentlength % 2880) {
  368       snprintf(errorstr,MAXLEN,"Content-Length not a multiple of 2880 (http_open) %d",
  369           contentlength);
  370       ffpmsg(errorstr);
  371     }
  372 
  373     /* write a memory file */
  374     alarm(net_timeout);
  375     while(0 != (len = fread(recbuf,1,MAXLEN,httpfile))) {
  376       alarm(0); /* cancel alarm */
  377       status = mem_write(*handle,recbuf,len);
  378       if (status) {
  379         ffpmsg("Error copying http file into memory (http_open)");
  380         ffpmsg(filename);
  381     goto error;
  382       }
  383       alarm(net_timeout); /* rearm the alarm */
  384     }
  385   }
  386   
  387   fclose(httpfile);
  388 
  389   signal(SIGALRM, SIG_DFL);
  390   alarm(0);
  391   return mem_seek(*handle,0);
  392 
  393  error:
  394   alarm(0); /* clear it */
  395   if (closehttpfile) {
  396     fclose(httpfile);
  397   }
  398   if (closememfile) {
  399     mem_close_free(*handle);
  400   }
  401   
  402   signal(SIGALRM, SIG_DFL);
  403   return (FILE_NOT_OPENED);
  404 }
  405 
  406 /*--------------------------------------------------------------------------*/
  407 /* This creates a memory file handle with a copy of the URL in filename.  The
  408    file must be compressed and is copied (still compressed) to disk first. 
  409    The compressed disk file is then uncompressed into memory (READONLY).
  410 */
  411 
  412 int http_compress_open(char *url, int rwmode, int *handle)
  413 {
  414   FILE *httpfile;
  415   char contentencoding[SHORTLEN];
  416   char errorstr[MAXLEN];
  417   char recbuf[MAXLEN];
  418   long len;
  419   int contentlength;
  420   int ii, flen, status;
  421   char firstchar;
  422 
  423   closehttpfile = 0;
  424   closediskfile = 0;
  425   closefdiskfile = 0;
  426   closememfile = 0;
  427 
  428   flen = strlen(netoutfile);
  429   if (!flen)  {
  430      /* cfileio made a mistake, should set the netoufile first otherwise 
  431         we don't know where to write the output file */
  432      ffpmsg
  433     ("Output file not set, shouldn't have happened (http_compress_open)");
  434       goto error;
  435   }
  436 
  437   if (rwmode != 0) {
  438     ffpmsg("Can't open compressed http:// type file with READWRITE access");
  439     ffpmsg("  Specify an UNCOMPRESSED outfile (http_compress_open)");
  440     goto error;
  441   }
  442   /* do the signal handler bits */
  443   if (setjmp(env) != 0) {
  444     /* feels like the second time */
  445     /* this means something bad happened */
  446     ffpmsg("Timeout (http_open)");
  447     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
  448     ffpmsg(errorstr);
  449     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
  450     goto error;
  451   }
  452 
  453   signal(SIGALRM, signal_handler);
  454   
  455   /* Open the http connectin */
  456   alarm(net_timeout);
  457   if ((status = http_open_network(url,&httpfile,contentencoding,
  458                    &contentlength))) {
  459     alarm(0);
  460     ffpmsg("Unable to open http file (http_compress_open)");
  461     ffpmsg(url);
  462     goto error;
  463   }
  464 
  465   closehttpfile++;
  466 
  467   /* Better be compressed */
  468 
  469   firstchar = fgetc(httpfile);
  470   ungetc(firstchar,httpfile);
  471   if (!strcmp(contentencoding,"x-gzip") || 
  472       !strcmp(contentencoding,"x-compress") ||
  473       ('\037' == firstchar)) {
  474 
  475     if (*netoutfile == '!')
  476     {
  477        /* user wants to clobber file, if it already exists */
  478        for (ii = 0; ii < flen; ii++)
  479            netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
  480 
  481        status = file_remove(netoutfile);
  482     }
  483 
  484     /* Create the new file */
  485     if ((status =  file_create(netoutfile,handle))) {
  486       ffpmsg("Unable to create output disk file (http_compress_open):");
  487       ffpmsg(netoutfile);
  488       goto error;
  489     }
  490     
  491     closediskfile++;
  492 
  493     /* write a file */
  494     alarm(net_timeout);
  495     while(0 != (len = fread(recbuf,1,MAXLEN,httpfile))) {
  496       alarm(0);
  497       status = file_write(*handle,recbuf,len);
  498       if (status) {
  499     ffpmsg("Error writing disk file (http_compres_open)");
  500         ffpmsg(netoutfile);
  501     goto error;
  502       }
  503       alarm(net_timeout);
  504     }
  505     file_close(*handle);
  506     fclose(httpfile);
  507     closehttpfile--;
  508     closediskfile--;
  509 
  510     /* File is on disk, let's uncompress it into memory */
  511 
  512     if (NULL == (diskfile = fopen(netoutfile,"r"))) {
  513       ffpmsg("Unable to reopen disk file (http_compress_open)");
  514       ffpmsg(netoutfile);
  515       goto error;
  516     }
  517     closefdiskfile++;
  518 
  519     /* Create the memory handle to hold it */
  520     if ((status =  mem_create(url,handle))) {
  521       ffpmsg("Unable to create memory file (http_compress_open)");
  522       goto error;
  523     }
  524     closememfile++;
  525 
  526     /* Uncompress it */
  527     status = 0;
  528     status = mem_uncompress2mem(url,diskfile,*handle);
  529     fclose(diskfile);
  530     closefdiskfile--;
  531     if (status) {
  532       ffpmsg("Error uncompressing disk file to memory (http_compress_open)");
  533       ffpmsg(netoutfile);
  534       goto error;
  535     }
  536       
  537   } else {
  538     /* Opps, this should not have happened */
  539     ffpmsg("Can only have compressed files here (http_compress_open)");
  540     goto error;
  541   }    
  542     
  543   signal(SIGALRM, SIG_DFL);
  544   alarm(0);
  545   return mem_seek(*handle,0);
  546 
  547  error:
  548   alarm(0); /* clear it */
  549   if (closehttpfile) {
  550     fclose(httpfile);
  551   }
  552   if (closefdiskfile) {
  553     fclose(diskfile);
  554   }
  555   if (closememfile) {
  556     mem_close_free(*handle);
  557   }
  558   if (closediskfile) {
  559     file_close(*handle);
  560   } 
  561   
  562   signal(SIGALRM, SIG_DFL);
  563   return (FILE_NOT_OPENED);
  564 }
  565 
  566 /*--------------------------------------------------------------------------*/
  567 /* This creates a file handle with a copy of the URL in filename.  The http
  568    file is copied to disk first.  If it's compressed then it is
  569    uncompressed when copying to the disk */
  570 
  571 int http_file_open(char *url, int rwmode, int *handle)
  572 {
  573   FILE *httpfile;
  574   char contentencoding[SHORTLEN];
  575   char errorstr[MAXLEN];
  576   char recbuf[MAXLEN];
  577   long len;
  578   int contentlength;
  579   int ii, flen, status;
  580   char firstchar;
  581 
  582   /* Check if output file is actually a memory file */
  583   if (!strncmp(netoutfile, "mem:", 4) )
  584   {
  585      /* allow the memory file to be opened with write access */
  586      return( http_open(url, READONLY, handle) );
  587   }     
  588 
  589   closehttpfile = 0;
  590   closefile = 0;
  591   closeoutfile = 0;
  592 
  593   flen = strlen(netoutfile);
  594   if (!flen) {
  595       /* cfileio made a mistake, we need to know where to write the file */
  596       ffpmsg("Output file not set, shouldn't have happened (http_file_open)");
  597       return (FILE_NOT_OPENED);
  598   }
  599 
  600   /* do the signal handler bits */
  601   if (setjmp(env) != 0) {
  602     /* feels like the second time */
  603     /* this means something bad happened */
  604     ffpmsg("Timeout (http_open)");
  605     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
  606     ffpmsg(errorstr);
  607     ffpmsg("   (multiplied x10 for files requiring uncompression)");
  608     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
  609     goto error;
  610   }
  611 
  612   signal(SIGALRM, signal_handler);
  613   
  614   /* Open the network connection */
  615   alarm(net_timeout);
  616   if ((status = http_open_network(url,&httpfile,contentencoding,
  617                    &contentlength))) {
  618     alarm(0);
  619     ffpmsg("Unable to open http file (http_file_open)");
  620     ffpmsg(url);
  621     goto error;
  622   }
  623 
  624   closehttpfile++;
  625 
  626   if (*netoutfile == '!')
  627   {
  628      /* user wants to clobber disk file, if it already exists */
  629      for (ii = 0; ii < flen; ii++)
  630          netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
  631 
  632      status = file_remove(netoutfile);
  633   }
  634 
  635   firstchar = fgetc(httpfile);
  636   ungetc(firstchar,httpfile);
  637   if (!strcmp(contentencoding,"x-gzip") || 
  638       !strcmp(contentencoding,"x-compress") ||
  639       ('\037' == firstchar)) {
  640 
  641     /* to make this more cfitsioish we use the file driver calls to create
  642        the disk file */
  643 
  644     /* Create the output file */
  645     if ((status =  file_create(netoutfile,handle))) {
  646       ffpmsg("Unable to create output file (http_file_open)");
  647       ffpmsg(netoutfile);
  648       goto error;
  649     }
  650 
  651     file_close(*handle);
  652     if (NULL == (outfile = fopen(netoutfile,"w"))) {
  653       ffpmsg("Unable to reopen the output file (http_file_open)");
  654       ffpmsg(netoutfile);
  655       goto error;
  656     }
  657     closeoutfile++;
  658     status = 0;
  659 
  660     /* Ok, this is a tough case, let's be arbritary and say 10*net_timeout,
  661        Given the choices for nettimeout above they'll probaby ^C before, but
  662        it's always worth a shot*/
  663 
  664     alarm(net_timeout*10);
  665     status = uncompress2file(url,httpfile,outfile,&status);
  666     alarm(0);
  667     if (status) {
  668       ffpmsg("Error uncompressing http file to disk file (http_file_open)");
  669       ffpmsg(url);
  670       ffpmsg(netoutfile);
  671       goto error;
  672     }
  673     fclose(outfile);
  674     closeoutfile--;
  675   } else {
  676     
  677     /* Create the output file */
  678     if ((status =  file_create(netoutfile,handle))) {
  679       ffpmsg("Unable to create output file (http_file_open)");
  680       ffpmsg(netoutfile);
  681       goto error;
  682     }
  683     
  684     /* Give a warning message.  This could just be bad padding at the end
  685        so don't treat it like an error. */
  686     closefile++;
  687     
  688     if (contentlength % 2880) {
  689       snprintf(errorstr, MAXLEN,
  690           "Content-Length not a multiple of 2880 (http_file_open) %d",
  691           contentlength);
  692       ffpmsg(errorstr);
  693     }
  694     
  695     /* write a file */
  696     alarm(net_timeout);
  697     while(0 != (len = fread(recbuf,1,MAXLEN,httpfile))) {
  698       alarm(0);
  699       status = file_write(*handle,recbuf,len);
  700       if (status) {
  701     ffpmsg("Error copying http file to disk file (http_file_open)");
  702         ffpmsg(url);
  703         ffpmsg(netoutfile);
  704     goto error;
  705       }
  706     }
  707     file_close(*handle);
  708     closefile--;
  709   }
  710   
  711   fclose(httpfile);
  712   closehttpfile--;
  713 
  714   signal(SIGALRM, SIG_DFL);
  715   alarm(0);
  716 
  717   return file_open(netoutfile,rwmode,handle); 
  718 
  719  error:
  720   alarm(0); /* clear it */
  721   if (closehttpfile) {
  722     fclose(httpfile);
  723   }
  724   if (closeoutfile) {
  725     fclose(outfile);
  726   }
  727   if (closefile) {
  728     file_close(*handle);
  729   } 
  730   
  731   signal(SIGALRM, SIG_DFL);
  732   return (FILE_NOT_OPENED);
  733 }
  734 
  735 /*--------------------------------------------------------------------------*/
  736 /* This is the guts of the code to get a file via http.  
  737    url is the input url
  738    httpfile is set to be the file connected to the socket which you can
  739      read the file from
  740    contentencoding is the mime type of the file, returned if the http server
  741      returns it
  742    contentlength is the length of the file, returned if the http server returns
  743      it
  744 */
  745 static int http_open_network(char *url, FILE **httpfile, char *contentencoding,
  746               int *contentlength)
  747 {
  748 
  749   int status;
  750   int sock;
  751   int tmpint;
  752   char recbuf[MAXLEN];
  753   char tmpstr[MAXLEN];
  754   char tmpstr1[SHORTLEN];
  755   char tmpstr2[MAXLEN];
  756   char errorstr[MAXLEN];
  757   char proto[SHORTLEN];
  758   char host[SHORTLEN];
  759   char userpass[MAXLEN];
  760   char fn[MAXLEN];
  761   char turl[MAXLEN];
  762   char *scratchstr;
  763   char *scratchstr2;
  764   char *saveptr;
  765   int port;
  766   float version;
  767 
  768   char pproto[SHORTLEN];
  769   char phost[SHORTLEN]; /* address of the proxy server */
  770   int  pport;  /* port number of the proxy server */
  771   char pfn[MAXLEN];
  772   char *proxy; /* URL of the proxy server */
  773 
  774   /* Parse the URL apart again */
  775   strcpy(turl,"http://");
  776   strncat(turl,url,MAXLEN - 8);
  777   if (NET_ParseUrl(turl,proto,host,&port,fn)) {
  778     snprintf(errorstr,MAXLEN,"URL Parse Error (http_open) %s",url);
  779     ffpmsg(errorstr);
  780     return (FILE_NOT_OPENED);
  781   }
  782 
  783   /* Do we have a user:password combo ? */
  784     strcpy(userpass, url);
  785   if ((scratchstr = strchr(userpass, '@')) != NULL) {
  786     *scratchstr = '\0';
  787   } else {
  788     strcpy(userpass, "");
  789   }
  790 
  791   /* Ph. Prugniel 2003/04/03
  792      Are we using a proxy?
  793      
  794      We use a proxy if the environment variable "http_proxy" is set to an
  795      address, eg. http://wwwcache.nottingham.ac.uk:3128
  796      ("http_proxy" is also used by wget)
  797   */
  798   proxy = getenv("http_proxy");
  799 
  800   /* Connect to the remote host */
  801   if (proxy) {
  802     if (NET_ParseUrl(proxy,pproto,phost,&pport,pfn)) {
  803       snprintf(errorstr,MAXLEN,"URL Parse Error (http_open) %s",proxy);
  804       ffpmsg(errorstr);
  805       return (FILE_NOT_OPENED);
  806     }
  807     sock = NET_TcpConnect(phost,pport);
  808   }  else {
  809     sock = NET_TcpConnect(host,port); 
  810   }
  811 
  812   if (sock < 0) {
  813     if (proxy) {
  814       ffpmsg("Couldn't connect to host via proxy server (http_open_network)");
  815       ffpmsg(proxy);
  816     }
  817     return (FILE_NOT_OPENED);
  818   }
  819 
  820   /* Make the socket a stdio file */
  821   if (NULL == (*httpfile = fdopen(sock,"r"))) {
  822     ffpmsg ("fdopen failed to convert socket to file (http_open_network)");
  823     close(sock);
  824     return (FILE_NOT_OPENED);
  825   }
  826 
  827   /* Send the GET request to the remote server */
  828   /* Ph. Prugniel 2003/04/03 
  829      One must add the Host: command because of HTTP 1.1 servers (ie. virtual
  830      hosts) */
  831 
  832   if (proxy) {
  833     snprintf(tmpstr,MAXLEN,"GET http://%s:%-d%s HTTP/1.0\r\n",host,port,fn);
  834   } else {
  835     snprintf(tmpstr,MAXLEN,"GET %s HTTP/1.0\r\n",fn);
  836   }
  837 
  838   if (strcmp(userpass, "")) {
  839     encode64(strlen(userpass), userpass, MAXLEN, tmpstr2);
  840     snprintf(tmpstr1, SHORTLEN,"Authorization: Basic %s\r\n", tmpstr2);
  841 
  842     if (strlen(tmpstr) + strlen(tmpstr1) > MAXLEN - 1)
  843     {
  844         fclose(*httpfile);
  845         *httpfile=0;
  846         return (FILE_NOT_OPENED);
  847     }
  848     strcat(tmpstr,tmpstr1);
  849   }
  850 
  851 /*  snprintf(tmpstr1,SHORTLEN,"User-Agent: HEASARC/CFITSIO/%-8.3f\r\n",ffvers(&version)); */
  852 
  853 /*  snprintf(tmpstr1,SHORTLEN,"User-Agent: CFITSIO/HEASARC/%-8.3f\r\n",ffvers(&version)); */
  854   snprintf(tmpstr1,SHORTLEN,"User-Agent: FITSIO/HEASARC/%-8.3f\r\n",ffvers(&version)); 
  855  
  856   if (strlen(tmpstr) + strlen(tmpstr1) > MAXLEN - 1)
  857   {
  858         fclose(*httpfile);
  859         *httpfile=0;
  860         return (FILE_NOT_OPENED);
  861   }
  862 
  863   strcat(tmpstr,tmpstr1);
  864 
  865   /* HTTP 1.1 servers require the following 'Host: ' string */
  866   snprintf(tmpstr1,SHORTLEN,"Host: %s:%-d\r\n\r\n",host,port);
  867 
  868   if (strlen(tmpstr) + strlen(tmpstr1) > MAXLEN - 1)
  869   {
  870         fclose(*httpfile);
  871         *httpfile=0;
  872         return (FILE_NOT_OPENED);
  873   }
  874 
  875   strcat(tmpstr,tmpstr1);
  876 
  877   status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
  878 
  879   /* read the header */
  880   if (!(fgets(recbuf,MAXLEN,*httpfile))) {
  881     snprintf (errorstr,MAXLEN,"http header short (http_open_network) %s",recbuf);
  882     ffpmsg(errorstr);
  883     fclose(*httpfile);
  884     *httpfile=0;
  885     return (FILE_NOT_OPENED);
  886   }
  887 
  888   *contentlength = 0;
  889   contentencoding[0] = '\0';
  890 
  891   /* Our choices are 200, ok, 302, temporary redirect, or 301 perm redirect */
  892   sscanf(recbuf,"%s %d",tmpstr,&status);
  893   if (status != 200){
  894     if (status == 301 || status == 302) {
  895       /* got a redirect */
  896 
  897 /*
  898       if (status == 302) {
  899     ffpmsg("Note: Web server replied with a temporary redirect from");
  900       } else {
  901     ffpmsg("Note: Web server replied with a redirect from");
  902       }
  903       ffpmsg(turl);
  904 */
  905       /* now, let's not write the most sophisticated parser here */
  906 
  907       while (fgets(recbuf,MAXLEN,*httpfile)) {
  908 
  909     scratchstr = strstr(recbuf,"Location: ");
  910     if (scratchstr != NULL) {
  911 
  912       /* Ok, we found the Location line which gives the redirected URL */
  913           /* skip the "Location: "  charactrers */
  914       scratchstr += 10; 
  915              
  916       /* strip off any end-of-line characters */
  917           tmpint = strlen(scratchstr);
  918       if (scratchstr[tmpint-1] == '\r') scratchstr[tmpint-1] = '\0';
  919           tmpint = strlen(scratchstr);
  920           if (scratchstr[tmpint-1] == '\n') scratchstr[tmpint-1] = '\0';
  921           tmpint = strlen(scratchstr);
  922       if (scratchstr[tmpint-1] == '\r') scratchstr[tmpint-1] = '\0';
  923 
  924 /*
  925       ffpmsg("to:");
  926       ffpmsg(scratchstr);
  927       ffpmsg(" ");
  928 */
  929       scratchstr2 = strstr(scratchstr,"http://");
  930           if (scratchstr2 != NULL) {
  931          /* Ok, we found the HTTP redirection is to another HTTP URL. */
  932          /* We can handle this case directly, here */
  933          /* skip the "http://" characters */
  934          scratchstr2 += 7;
  935          strcpy(turl, scratchstr2);
  936          fclose (*httpfile);
  937              *httpfile=0;
  938 
  939              /* note the recursive call to itself */
  940          return 
  941             http_open_network(turl,httpfile,contentencoding,contentlength);
  942           }
  943 
  944           /* It was not a HTTP to HTTP redirection, so see if it HTTP to FTP */
  945       scratchstr2 = strstr(scratchstr,"ftp://");
  946           if (scratchstr2 != NULL) {
  947          /* Ok, we found the HTTP redirection is to a FTP URL. */
  948          /* skip the "ftp://" characters */
  949          scratchstr2 += 6;
  950 
  951              /* return the new URL string, and set contentencoding to "ftp" as
  952             a flag to the http_checkfile routine
  953          */
  954              if (strlen(scratchstr2) > FLEN_FILENAME-1) 
  955              {
  956                 ffpmsg("Error: redirected url string too long (http_open_network)");
  957                 fclose(*httpfile);
  958                 *httpfile=0;
  959                 return URL_PARSE_ERROR;
  960              }
  961          strcpy(url, scratchstr2);
  962              strcpy(contentencoding,"ftp://");
  963          fclose (*httpfile);
  964              *httpfile=0; 
  965          return 0;
  966           }
  967           
  968           /* Now check for HTTP to HTTPS redirection. */
  969       scratchstr2 = strstr(scratchstr,"https://");
  970           if (scratchstr2 != NULL) {
  971              /* skip the "https://" characters */
  972              scratchstr2 += 8;
  973              
  974              /* return the new URL string, and set contentencoding to "https" as
  975             a flag to the http_checkfile routine
  976          */
  977              if (strlen(scratchstr2) > FLEN_FILENAME-1) 
  978              {
  979                 ffpmsg("Error: redirected url string too long (http_open_network)");
  980                 fclose(*httpfile);
  981                 return URL_PARSE_ERROR;
  982              }
  983              strcpy(url, scratchstr2);
  984              strcpy(contentencoding,"https://");
  985              fclose(*httpfile);
  986              *httpfile=0;
  987              return 0;
  988           }
  989           
  990     }
  991       }
  992 
  993       /* if we get here then we couldnt' decide the redirect */
  994       ffpmsg("but we were unable to find the redirected url in the servers response");
  995     }
  996 
  997     /* error.  could not open the http file */
  998     fclose(*httpfile);
  999     *httpfile=0;
 1000     return (FILE_NOT_OPENED);
 1001   }
 1002 
 1003   /* from here the first word holds the keyword we want */
 1004   /* so, read the rest of the header */
 1005   while (fgets(recbuf,MAXLEN,*httpfile)) {
 1006     /* Blank line ends the header */
 1007     if (*recbuf == '\r') break;
 1008     if (strlen(recbuf) > 3) {
 1009       recbuf[strlen(recbuf)-1] = '\0';
 1010       recbuf[strlen(recbuf)-1] = '\0';
 1011     }
 1012     sscanf(recbuf,"%s %d",tmpstr,&tmpint);
 1013     /* Did we get a content-length header ? */
 1014     if (!strcmp(tmpstr,"Content-Length:")) {
 1015       *contentlength = tmpint;
 1016     }
 1017     /* Did we get the content-encoding header ? */
 1018     if (!strcmp(tmpstr,"Content-Encoding:")) {
 1019       if (NULL != (scratchstr = strstr(recbuf,":"))) {
 1020     /* Found the : */
 1021     scratchstr++; /* skip the : */
 1022     scratchstr++; /* skip the extra space */
 1023         if (strlen(scratchstr) > SHORTLEN-1) 
 1024         {
 1025            ffpmsg("Error: content-encoding string too long (http_open_network)");
 1026            fclose(*httpfile);
 1027            *httpfile=0;
 1028            return URL_PARSE_ERROR;
 1029         }
 1030     strcpy(contentencoding,scratchstr);
 1031       }
 1032     }
 1033   }
 1034   
 1035   /* we're done, so return */
 1036   return 0;
 1037 }
 1038 
 1039 /*--------------------------------------------------------------------------*/
 1040 /* This creates a memory file handle with a copy of the URL in filename. The 
 1041    curl library called from https_open_network will perform file uncompression
 1042    if necessary. */
 1043 int https_open(char *filename, int rwmode, int *handle)
 1044 {
 1045   curlmembuf inmem;
 1046   char errStr[MAXLEN];
 1047   int status=0;
 1048     
 1049   /* don't do r/w files */
 1050   if (rwmode != 0) {
 1051     ffpmsg("Can't open https:// type file with READWRITE access");
 1052     ffpmsg("  Specify an outfile for r/w access (https_open)");
 1053     return (FILE_NOT_OPENED);
 1054   }
 1055 
 1056   inmem.memory=0;
 1057   inmem.size=0;
 1058   if (setjmp(env) != 0)
 1059   {
 1060     alarm(0);
 1061     signal(SIGALRM, SIG_DFL);
 1062     ffpmsg("Timeout (https_open)");
 1063     snprintf(errStr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 1064     ffpmsg(errStr);
 1065     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 1066     free(inmem.memory);
 1067     return (FILE_NOT_OPENED);
 1068   }
 1069 
 1070   signal(SIGALRM, signal_handler);
 1071   alarm(net_timeout);
 1072 
 1073   if (https_open_network(filename, &inmem))
 1074   {
 1075      alarm(0);
 1076      signal(SIGALRM, SIG_DFL);
 1077      ffpmsg("Unable to read https file into memory (https_open)");
 1078      free(inmem.memory);
 1079      return (FILE_NOT_OPENED);  
 1080   }
 1081   alarm(0);
 1082   signal(SIGALRM, SIG_DFL);
 1083   /* We now have the file transfered from the https server into the
 1084      inmem.memory buffer.  Now transfer that into a FITS memory file. */
 1085   if ((status = mem_create(filename, handle)))
 1086   {
 1087      ffpmsg("Unable to create memory file (https_open)");
 1088      free(inmem.memory);
 1089      return (FILE_NOT_OPENED);
 1090   }
 1091   
 1092   if (inmem.size % 2880)
 1093   {
 1094      snprintf(errStr,MAXLEN,"Content-Length not a multiple of 2880 (https_open) %u",
 1095          inmem.size);
 1096      ffpmsg(errStr);
 1097   }
 1098   status = mem_write(*handle, inmem.memory, inmem.size);
 1099   if (status)
 1100   {
 1101      ffpmsg("Error copying https file into memory (https_open)");
 1102      ffpmsg(filename);
 1103      free(inmem.memory);
 1104      mem_close_free(*handle);
 1105      return (FILE_NOT_OPENED);
 1106   }
 1107   free(inmem.memory);
 1108   return mem_seek(*handle, 0);
 1109    
 1110 }
 1111 
 1112 /*--------------------------------------------------------------------------*/
 1113 int https_file_open(char *filename, int rwmode, int *handle)
 1114 {
 1115   int ii, flen;
 1116   char errStr[MAXLEN];
 1117   curlmembuf inmem;
 1118   
 1119   /* Check if output file is actually a memory file */
 1120   if (!strncmp(netoutfile, "mem:", 4) )
 1121   {
 1122      /* allow the memory file to be opened with write access */
 1123      return( https_open(filename, READONLY, handle) );
 1124   }     
 1125 
 1126   flen = strlen(netoutfile);
 1127   if (!flen)
 1128   {
 1129       /* cfileio made a mistake, we need to know where to write the file */
 1130       ffpmsg("Output file not set, shouldn't have happened (https_file_open)");
 1131       return (FILE_NOT_OPENED);
 1132   }
 1133   
 1134   inmem.memory=0;
 1135   inmem.size=0;
 1136   if (setjmp(env) != 0)
 1137   {
 1138      alarm(0);
 1139      signal(SIGALRM, SIG_DFL);
 1140      ffpmsg("Timeout (https_file_open)");
 1141      snprintf(errStr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 1142      ffpmsg(errStr);
 1143      ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 1144      free(inmem.memory);
 1145      return (FILE_NOT_OPENED);
 1146   }
 1147   signal(SIGALRM, signal_handler);
 1148   alarm(net_timeout);
 1149   if (https_open_network(filename, &inmem))
 1150   {
 1151      alarm(0);
 1152      signal(SIGALRM, SIG_DFL);
 1153      ffpmsg("Unable to read https file into memory (https_file_open)");
 1154      free(inmem.memory);
 1155      return (FILE_NOT_OPENED);  
 1156   }
 1157   alarm(0);
 1158   signal(SIGALRM, SIG_DFL);
 1159   
 1160   if (*netoutfile == '!')
 1161   {
 1162      /* user wants to clobber disk file, if it already exists */
 1163      for (ii = 0; ii < flen; ii++)
 1164          netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
 1165 
 1166      file_remove(netoutfile);
 1167   }
 1168 
 1169   /* Create the output file */
 1170   if (file_create(netoutfile,handle)) 
 1171   {
 1172     ffpmsg("Unable to create output file (https_file_open)");
 1173     ffpmsg(netoutfile);
 1174     free(inmem.memory);
 1175     return (FILE_NOT_OPENED);
 1176   }
 1177     
 1178   if (inmem.size % 2880)
 1179   {
 1180     snprintf(errStr, MAXLEN,
 1181         "Content-Length not a multiple of 2880 (https_file_open) %d",
 1182         inmem.size);
 1183     ffpmsg(errStr);
 1184   }
 1185    
 1186   if (file_write(*handle, inmem.memory, inmem.size))
 1187   {
 1188      ffpmsg("Error copying https file to disk file (https_file_open)");
 1189      ffpmsg(filename);
 1190      ffpmsg(netoutfile);
 1191      free(inmem.memory);
 1192      file_close(*handle);
 1193      return (FILE_NOT_OPENED);
 1194   }
 1195   free(inmem.memory); 
 1196   file_close(*handle);
 1197      
 1198   return file_open(netoutfile, rwmode, handle);
 1199 }
 1200 
 1201 /*--------------------------------------------------------------------------*/
 1202 /* Callback function curl library uses during https connection to transfer
 1203    server file into memory */
 1204 size_t curlToMemCallback(void *buffer, size_t size, size_t nmemb, void *userp)
 1205 {
 1206    curlmembuf* inmem = (curlmembuf* )userp;
 1207    size_t transferSize = size*nmemb;
 1208    if (!inmem->size)
 1209    {
 1210       /* First time through - initialize with malloc */
 1211       inmem->memory = (char *)malloc(transferSize); 
 1212    }
 1213    else
 1214       inmem->memory = realloc(inmem->memory, inmem->size+transferSize);
 1215    if (inmem->memory == NULL)
 1216    {
 1217       ffpmsg("realloc error - not enough memory (curlToMemCallback)\n");
 1218       return 0;
 1219    }
 1220    memcpy(&(inmem->memory[inmem->size]), buffer, transferSize);
 1221    inmem->size += transferSize;
 1222    
 1223    return transferSize;
 1224 }
 1225 
 1226 /*--------------------------------------------------------------------------*/
 1227 /* Callback function for displaying status bar during download */
 1228 int curlProgressCallback(void *clientp, double dltotal, double dlnow,
 1229       double ultotal, double ulnow)
 1230 {
 1231    int i, fullBar = 50, nToDisplay = 0;
 1232    int percent = 0;
 1233    double fracCompleted = 0.0;
 1234    char *urlname=0;
 1235    static int isComplete = 0;
 1236    static int isFirst = 1;
 1237    
 1238    /* isFirst is true the very first time this is entered. Afterwards it
 1239       should get reset to true when isComplete is first detected to have 
 1240       toggled from true to false. */
 1241    if (dltotal == 0.0)
 1242    {
 1243       if (isComplete)
 1244          isFirst = 1;
 1245       isComplete = 0;
 1246       return 0;
 1247    }
 1248 
 1249    fracCompleted = dlnow/dltotal;
 1250    percent = (int)ceil(fracCompleted*100.0 - 0.5);
 1251    if (isComplete && percent < 100)
 1252       isFirst = 1;
 1253    if (!isComplete || percent < 100)
 1254    {
 1255       if (isFirst)
 1256       {
 1257          urlname = (char *)clientp;
 1258          if (urlname)
 1259          {
 1260             fprintf(stderr,"Downloading ");
 1261             fprintf(stderr,"%s",urlname);
 1262             fprintf(stderr,"...\n");
 1263          }
 1264          isFirst = 0;
 1265       }
 1266       isComplete = (percent >= 100) ? 1 : 0;
 1267       nToDisplay = (int)ceil(fracCompleted*fullBar - 0.5);
 1268       /* Can dlnow ever be > dltotal?  Just in case... */
 1269       if (nToDisplay > fullBar)
 1270          nToDisplay = fullBar;
 1271       fprintf(stderr,"%3d%% [",percent);
 1272       for (i=0; i<nToDisplay; ++i)
 1273          fprintf(stderr,"=");
 1274       /* print remaining spaces */
 1275       for (i=nToDisplay; i<fullBar; ++i)
 1276          fprintf(stderr," ");
 1277       fprintf(stderr,"]\r");
 1278       if (isComplete)
 1279          fprintf(stderr,"\n");
 1280       fflush(stderr);
 1281    }
 1282    return 0;
 1283 }
 1284 
 1285 /*--------------------------------------------------------------------------*/
 1286 int https_open_network(char *filename, curlmembuf* buffer)
 1287 {
 1288   int status=0;
 1289   char *urlname=0;
 1290   
 1291   /* urlname may have .gz or .Z appended to it */
 1292   urlname = (char *)malloc(strlen(filename)+12);
 1293   strcpy(urlname,"https://");
 1294   strcat(urlname,filename);
 1295   status = ssl_get_with_curl(urlname, buffer, 0, 0);
 1296   free(urlname);
 1297   return(status);
 1298 }
 1299 
 1300 void https_set_verbose(int flag)
 1301 {
 1302    if (!flag)
 1303       curl_verbose = 0;
 1304    else
 1305       curl_verbose = 1;
 1306 }
 1307 
 1308 void fits_dwnld_prog_bar(int flag)
 1309 {
 1310    if (!flag)
 1311       show_fits_download_progress = 0;
 1312    else
 1313       show_fits_download_progress = 1;
 1314 }
 1315 
 1316 int fits_net_timeout(int sec)
 1317 {
 1318    /* If sec is 0 or negative, treat this as a 'get' call. */
 1319    if (sec > 0)
 1320       net_timeout = (unsigned int)sec;
 1321    return (int)net_timeout;
 1322 }
 1323 
 1324 /*--------------------------------------------------------------------------*/
 1325 int ftps_open(char *filename, int rwmode, int *handle)
 1326 {
 1327   curlmembuf inmem;
 1328   char errStr[MAXLEN];
 1329   char localFilename[MAXLEN]; /* may have .gz or .Z appended in ftps_open_network.*/
 1330   unsigned char firstByte=0,secondByte=0;
 1331   int status=0;
 1332   FILE *compressedFile=0;
 1333   
 1334   strcpy(localFilename,filename);
 1335     
 1336   /* don't do r/w files */
 1337   if (rwmode != 0) {
 1338     ffpmsg("Can't open ftps:// type file with READWRITE access");
 1339     ffpmsg("  Specify an outfile for r/w access (ftps_open)");
 1340     return (FILE_NOT_OPENED);
 1341   }
 1342 
 1343   inmem.memory=0;
 1344   inmem.size=0;
 1345   if (setjmp(env) != 0)
 1346   {
 1347     alarm(0);
 1348     signal(SIGALRM, SIG_DFL);
 1349     ffpmsg("Timeout (ftps_open)");
 1350     snprintf(errStr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 1351     ffpmsg(errStr);
 1352     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 1353     free(inmem.memory);
 1354     return (FILE_NOT_OPENED);
 1355   }
 1356 
 1357   signal(SIGALRM, signal_handler);
 1358   alarm(net_timeout);
 1359 
 1360   if (ftps_open_network(localFilename, &inmem))
 1361   {
 1362      alarm(0);
 1363      signal(SIGALRM, SIG_DFL);
 1364      ffpmsg("Unable to read ftps file into memory (ftps_open)");
 1365      free(inmem.memory);
 1366      return (FILE_NOT_OPENED);  
 1367   }
 1368   
 1369   alarm(0);
 1370   signal(SIGALRM, SIG_DFL);
 1371 
 1372   if (strcmp(localFilename, filename))
 1373   {
 1374      /* ftps_open_network has already checked that this is safe to
 1375         copy into string of size FLEN_FILENAME */
 1376      strcpy(filename, localFilename);
 1377   }
 1378   
 1379   /* We now have the file transfered from the ftps server into the
 1380      inmem.memory buffer.  Now transfer that into a FITS memory file. */
 1381   if ((status = mem_create(filename, handle)))
 1382   {
 1383      ffpmsg("Unable to create memory file (ftps_open)");
 1384      free(inmem.memory);
 1385      return (FILE_NOT_OPENED);
 1386   }
 1387   if (inmem.size > 1)
 1388   {
 1389      firstByte = (unsigned char)inmem.memory[0];
 1390      secondByte = (unsigned char)inmem.memory[1];
 1391   }
 1392   if (firstByte == 0x1f && secondByte == 0x8b || 
 1393         strstr(localFilename,".Z"))
 1394   {
 1395 #ifdef HAVE_FMEMOPEN
 1396      compressedFile = fmemopen(inmem.memory, inmem.size, "r");
 1397 #endif
 1398      if (!compressedFile)
 1399      {
 1400         ffpmsg("Error creating file in memory (ftps_open)");
 1401         free(inmem.memory);
 1402         return(FILE_NOT_OPENED);
 1403      }
 1404      if(mem_uncompress2mem(localFilename,compressedFile,*handle))
 1405      {
 1406         ffpmsg("Error writing compressed memory file (ftps_open)");
 1407         ffpmsg(filename);
 1408         fclose(compressedFile);
 1409         free(inmem.memory);
 1410         return(FILE_NOT_OPENED);
 1411      }
 1412      fclose(compressedFile);
 1413   }
 1414   else
 1415   {
 1416      if (inmem.size % 2880)
 1417      {
 1418         snprintf(errStr,MAXLEN,"Content-Length not a multiple of 2880 (ftps_open) %u",
 1419             inmem.size);
 1420         ffpmsg(errStr);
 1421      }
 1422      status = mem_write(*handle, inmem.memory, inmem.size);
 1423      if (status)
 1424      {
 1425         ffpmsg("Error copying https file into memory (ftps_open)");
 1426         ffpmsg(filename);
 1427         free(inmem.memory);
 1428         mem_close_free(*handle);
 1429         return (FILE_NOT_OPENED);
 1430      }
 1431   }
 1432   free(inmem.memory);
 1433   return mem_seek(*handle, 0);
 1434 }
 1435 
 1436 /*--------------------------------------------------------------------------*/
 1437 int ftps_file_open(char *filename, int rwmode, int *handle)
 1438 {
 1439   int ii, flen, status=0;
 1440   char errStr[MAXLEN];
 1441   char localFilename[MAXLEN]; /* may have .gz or .Z appended */
 1442   unsigned char firstByte=0,secondByte=0;
 1443   curlmembuf inmem;
 1444   FILE *compressedInFile=0;
 1445   
 1446   strcpy(localFilename, filename);
 1447   
 1448   /* Check if output file is actually a memory file */
 1449   if (!strncmp(netoutfile, "mem:", 4) )
 1450   {
 1451      /* allow the memory file to be opened with write access */
 1452      return( ftps_open(filename, READONLY, handle) );
 1453   }     
 1454 
 1455   flen = strlen(netoutfile);
 1456   if (!flen)
 1457   {
 1458       /* cfileio made a mistake, we need to know where to write the file */
 1459       ffpmsg("Output file not set, shouldn't have happened (ftps_file_open)");
 1460       return (FILE_NOT_OPENED);
 1461   }
 1462   
 1463   inmem.memory=0;
 1464   inmem.size=0;
 1465   if (setjmp(env) != 0)
 1466   {
 1467      alarm(0);
 1468      signal(SIGALRM, SIG_DFL);
 1469      ffpmsg("Timeout (ftps_file_open)");
 1470      snprintf(errStr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 1471      ffpmsg(errStr);
 1472      ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 1473      free(inmem.memory);
 1474      return (FILE_NOT_OPENED);
 1475   }
 1476   signal(SIGALRM, signal_handler);
 1477   alarm(net_timeout);
 1478   if (ftps_open_network(localFilename, &inmem))
 1479   {
 1480      alarm(0);
 1481      signal(SIGALRM, SIG_DFL);
 1482      ffpmsg("Unable to read ftps file into memory (ftps_file_open)");
 1483      free(inmem.memory);
 1484      return (FILE_NOT_OPENED);  
 1485   }
 1486   alarm(0);
 1487   signal(SIGALRM, SIG_DFL);
 1488   
 1489   if (strstr(localFilename, ".Z"))
 1490   {
 1491      ffpmsg(".Z decompression not supported for file output (ftps_file_open)");
 1492      free(inmem.memory);
 1493      return (FILE_NOT_OPENED);
 1494   }
 1495   
 1496   if (strcmp(localFilename, filename))
 1497   {
 1498      /* ftps_open_network has already checked that this is safe to
 1499         copy into string of size FLEN_FILENAME */
 1500      strcpy(filename, localFilename);
 1501   }
 1502   
 1503   if (*netoutfile == '!')
 1504   {
 1505      /* user wants to clobber disk file, if it already exists */
 1506      for (ii = 0; ii < flen; ii++)
 1507          netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
 1508 
 1509      file_remove(netoutfile);
 1510   }
 1511 
 1512   /* Create the output file */
 1513   if (file_create(netoutfile,handle)) 
 1514   {
 1515     ffpmsg("Unable to create output file (ftps_file_open)");
 1516     ffpmsg(netoutfile);
 1517     free(inmem.memory);
 1518     return (FILE_NOT_OPENED);
 1519   }
 1520   
 1521   if (inmem.size > 1)
 1522   {  
 1523      firstByte = (unsigned char)inmem.memory[0];
 1524      secondByte = (unsigned char)inmem.memory[1];
 1525   }
 1526   if (firstByte == 0x1f && secondByte == 0x8b)
 1527   {
 1528      /* Doing a file create/close/reopen to mimic the procedure in
 1529         ftp_file_open.  The earlier call to file_create ensures that 
 1530         checking is performed for the Hera case. */
 1531      file_close(*handle);
 1532      /* Reopen with direct call to fopen to set the outfile pointer */
 1533      outfile = fopen(netoutfile,"w");
 1534      if (!outfile)
 1535      {
 1536         ffpmsg("Unable to reopen the output file (ftps_file_open)");
 1537         ffpmsg(netoutfile);
 1538         free(inmem.memory);
 1539         return(FILE_NOT_OPENED);
 1540      }
 1541      
 1542 #ifdef HAVE_FMEMOPEN
 1543      compressedInFile = fmemopen(inmem.memory, inmem.size, "r");
 1544 #endif
 1545      if (!compressedInFile)
 1546      {
 1547         ffpmsg("Error creating compressed file in memory (ftps_file_open)");
 1548         free(inmem.memory);
 1549         fclose(outfile);
 1550         return(FILE_NOT_OPENED);
 1551      }
 1552      if (uncompress2file(filename, compressedInFile, outfile, &status))
 1553      {
 1554         ffpmsg("Unable to uncompress the output file (ftps_file_open)");
 1555         ffpmsg(filename);
 1556         ffpmsg(netoutfile);
 1557         fclose(outfile);
 1558         fclose(compressedInFile);
 1559         free(inmem.memory);
 1560         return(FILE_NOT_OPENED);
 1561      }
 1562      fclose(outfile);
 1563      fclose(compressedInFile);
 1564   }
 1565   else
 1566   {
 1567      if (inmem.size % 2880)
 1568      {
 1569        snprintf(errStr, MAXLEN,
 1570            "Content-Length not a multiple of 2880 (ftps_file_open) %d",
 1571            inmem.size);
 1572        ffpmsg(errStr);
 1573      }
 1574 
 1575      if (file_write(*handle, inmem.memory, inmem.size))
 1576      {
 1577         ffpmsg("Error copying ftps file to disk file (ftps_file_open)");
 1578         ffpmsg(filename);
 1579         ffpmsg(netoutfile);
 1580         free(inmem.memory);
 1581         file_close(*handle);
 1582         return (FILE_NOT_OPENED);
 1583      }
 1584      file_close(*handle);
 1585   }
 1586   free(inmem.memory); 
 1587   
 1588   return file_open(netoutfile, rwmode, handle);
 1589   
 1590 }
 1591 
 1592 /*--------------------------------------------------------------------------*/
 1593 int ftps_compress_open(char *filename, int rwmode, int *handle)
 1594 {
 1595    int ii, flen, status=0;
 1596   char errStr[MAXLEN];
 1597   char localFilename[MAXLEN]; /* may have .gz or .Z appended */
 1598   unsigned char firstByte=0,secondByte=0;
 1599   curlmembuf inmem;
 1600   FILE *compressedInFile=0;
 1601   
 1602   /* don't do r/w files */
 1603   if (rwmode != 0) {
 1604     ffpmsg("Compressed files must be r/o");
 1605     return (FILE_NOT_OPENED);
 1606   }
 1607   
 1608   strcpy(localFilename, filename);
 1609   
 1610   flen = strlen(netoutfile);
 1611   if (!flen)
 1612   {
 1613       /* cfileio made a mistake, we need to know where to write the file */
 1614       ffpmsg("Output file not set, shouldn't have happened (ftps_compress_open)");
 1615       return (FILE_NOT_OPENED);
 1616   }
 1617   
 1618   inmem.memory=0;
 1619   inmem.size=0;
 1620   if (setjmp(env) != 0)
 1621   {
 1622      alarm(0);
 1623      signal(SIGALRM, SIG_DFL);
 1624      ffpmsg("Timeout (ftps_compress_open)");
 1625      snprintf(errStr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 1626      ffpmsg(errStr);
 1627      ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 1628      free(inmem.memory);
 1629      return (FILE_NOT_OPENED);
 1630   }
 1631   signal(SIGALRM, signal_handler);
 1632   alarm(net_timeout);
 1633   if (ftps_open_network(localFilename, &inmem))
 1634   {
 1635      alarm(0);
 1636      signal(SIGALRM, SIG_DFL);
 1637      ffpmsg("Unable to read ftps file into memory (ftps_compress_open)");
 1638      free(inmem.memory);
 1639      return (FILE_NOT_OPENED);  
 1640   }
 1641   alarm(0);
 1642   signal(SIGALRM, SIG_DFL);
 1643   
 1644   if (strcmp(localFilename, filename))
 1645   {
 1646      /* ftps_open_network has already checked that this is safe to
 1647         copy into string of size FLEN_FILENAME */
 1648      strcpy(filename, localFilename);
 1649   }
 1650   if (inmem.size > 1)
 1651   {  
 1652      firstByte = (unsigned char)inmem.memory[0];
 1653      secondByte = (unsigned char)inmem.memory[1];
 1654   }
 1655   if ((firstByte == 0x1f && secondByte == 0x8b) || 
 1656         strstr(localFilename,".gz") || strstr(localFilename,".Z"))
 1657   {
 1658      if (*netoutfile == '!')
 1659      {
 1660         /* user wants to clobber disk file, if it already exists */
 1661         for (ii = 0; ii < flen; ii++)
 1662             netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
 1663 
 1664         file_remove(netoutfile);
 1665      }
 1666      /* Create the output file */
 1667      if (file_create(netoutfile,handle)) 
 1668      {
 1669        ffpmsg("Unable to create output file (ftps_compress_open)");
 1670        ffpmsg(netoutfile);
 1671        free(inmem.memory);
 1672        return (FILE_NOT_OPENED);
 1673      }
 1674      if (file_write(*handle, inmem.memory, inmem.size))
 1675      {
 1676         ffpmsg("Error copying ftps file to disk file (ftps_file_open)");
 1677         ffpmsg(filename);
 1678         ffpmsg(netoutfile);
 1679         free(inmem.memory);
 1680         file_close(*handle);
 1681         return (FILE_NOT_OPENED);
 1682      }
 1683      file_close(*handle);
 1684 
 1685     /* File is on disk, let's uncompress it into memory */
 1686     if (NULL == (diskfile = fopen(netoutfile,"r"))) {
 1687       ffpmsg("Unable to reopen disk file (ftps_compress_open)");
 1688       ffpmsg(netoutfile);
 1689       free(inmem.memory);
 1690       return (FILE_NOT_OPENED);
 1691     }
 1692 
 1693     if ((status =  mem_create(localFilename,handle))) {
 1694       ffpmsg("Unable to create memory file (ftps_compress_open)");
 1695       ffpmsg(localFilename);
 1696       free(inmem.memory);
 1697       fclose(diskfile);
 1698       diskfile=0;
 1699       return (FILE_NOT_OPENED);
 1700     }
 1701 
 1702     status = mem_uncompress2mem(localFilename,diskfile,*handle);
 1703     fclose(diskfile);
 1704     diskfile=0;
 1705 
 1706     if (status) {
 1707       ffpmsg("Error writing compressed memory file (ftps_compress_open)");
 1708       free(inmem.memory);
 1709       mem_close_free(*handle);
 1710       return (FILE_NOT_OPENED);
 1711      }
 1712       
 1713   }
 1714   else
 1715   {
 1716      ffpmsg("Cannot write uncompressed infile to compressed outfile (ftps_compress_open)");
 1717      free(inmem.memory);
 1718      return (FILE_NOT_OPENED);
 1719   }
 1720       
 1721   free(inmem.memory); 
 1722   
 1723   return mem_seek(*handle,0);
 1724   
 1725 }
 1726 
 1727 /*--------------------------------------------------------------------------*/
 1728 int ftps_open_network(char *filename, curlmembuf* buffer)
 1729 {
 1730   char agentStr[SHORTLEN];
 1731   char url[MAXLEN];
 1732   char tmphost[SHORTLEN]; /* work array for separating user/pass/host names */
 1733   char *username=0;
 1734   char *password=0;
 1735   char *hostname=0;
 1736   char *dirpath=0;
 1737   char *strptr=0;
 1738   float version=0.0;
 1739   int iDirpath=0, len=0, origLen=0;
 1740   int status=0; 
 1741   
 1742   strcpy(url,"ftp://");
 1743 
 1744   /* The filename may already contain a username and password, as indicated 
 1745      by a '@' within the host part of the name (which we'll define as the substring
 1746      before the first '/').  If not, we'll set a default username:password  */
 1747   len = strlen(filename);
 1748   for (iDirpath=0; iDirpath<len; ++iDirpath)
 1749   {
 1750      if (filename[iDirpath] == '/')
 1751         break;
 1752   }
 1753   if (iDirpath > SHORTLEN-1)
 1754   {
 1755      ffpmsg("Host name is too long in URL (ftps_open_network)");
 1756      return (FILE_NOT_OPENED);
 1757   }
 1758   strncpy(tmphost, filename, iDirpath);
 1759   dirpath = &filename[iDirpath];
 1760   tmphost[iDirpath]='\0';
 1761   
 1762   /* There could be more than one '@' since they can also exist in the
 1763      username or password.  Find the right-most '@' and assume that it
 1764      delimits the host name. */
 1765   hostname = strrchr(tmphost, '@');
 1766   if (hostname)
 1767   {
 1768      *hostname = '\0';
 1769      ++hostname;
 1770      /* Assume first occurrence of ':' is indicative of password delimiter. */
 1771      password = strchr(tmphost, ':');
 1772      if (password)
 1773      {
 1774         *password = '\0';
 1775         ++password;
 1776      }
 1777      username = tmphost;
 1778   }
 1779   else
 1780      hostname = tmphost;
 1781   
 1782   if (!username || strlen(username)==0)
 1783      username = "anonymous";
 1784   if (!password || strlen(password)==0)
 1785   {
 1786      snprintf(agentStr,SHORTLEN,"User-Agent: FITSIO/HEASARC/%-8.3f",ffvers(&version));
 1787      password = agentStr;
 1788   }
 1789   
 1790   /* url may eventually have .gz or .Z appended to it */
 1791   if (strlen(url) + strlen(hostname) + strlen(dirpath) > MAXLEN-4)
 1792   {
 1793      ffpmsg("Full URL name is too long (ftps_open_network)");
 1794      return (FILE_NOT_OPENED);
 1795   }
 1796   strcat(url, hostname);
 1797   strcat(url, dirpath);
 1798   
 1799 /*  printf("url = %s\n",url);
 1800   printf("username = %s\n",username);
 1801   printf("password = %s\n",password);
 1802   printf("hostname = %s\n",hostname);
 1803 */
 1804 
 1805   origLen = strlen(url);
 1806   status = ssl_get_with_curl(url, buffer, username, password);
 1807   /* If original url has .gz or .Z appended, do the same to the original filename.
 1808      Note that url also differs from original filename at this point, since
 1809      filename may have included username@password (which url would not). */
 1810   len = strlen(url);
 1811   if ((len-origLen) == 2 || (len-origLen) == 3)
 1812   {
 1813      if (strlen(filename) > FLEN_FILENAME - 4)
 1814      {
 1815         ffpmsg("Filename is too long to append compression ext (ftps_open_network)");
 1816         /* buffer memory must be freed by calling routine */
 1817         return (FILE_NOT_OPENED);
 1818      }
 1819      strptr = url + origLen;
 1820      strcat(filename, strptr);
 1821   }
 1822   return status;
 1823   
 1824  }
 1825 
 1826 /*--------------------------------------------------------------------------*/
 1827 /* Function to perform common curl interfacing for https or ftps transfers */
 1828 
 1829 int ssl_get_with_curl(char *url, curlmembuf* buffer, char* username,
 1830                         char* password)
 1831 {
 1832   /* These settings will force libcurl to perform host and peer authentication.
 1833      If it fails, this routine will try again without authentication (unless
 1834      user forbids this via CFITSIO_VERIFY_HTTPS environment variable).
 1835   */
 1836   long verifyPeer = 1;
 1837   long verifyHost = 2;
 1838   char errStr[MAXLEN];
 1839   char agentStr[MAXLEN];
 1840   float version=0.0;
 1841   char *tmpUrl=0;
 1842   char *verify=0;
 1843   int isFtp = (strstr(url,"ftp://") != NULL);
 1844   int experimentWithCompression = (!strstr(url,".gz") && !strstr(url,".Z")
 1845                 && !strstr(url,"?"));
 1846   int notFound=1;
 1847   #ifdef CFITSIO_HAVE_CURL
 1848   CURL *curl=0;
 1849   CURLcode res;
 1850   char curlErrBuf[CURL_ERROR_SIZE];
 1851   
 1852   if (strstr(url,".Z") && !isFtp)
 1853   {
 1854      ffpmsg("x-compress .Z format not currently supported with curl https transfers");
 1855      return(FILE_NOT_OPENED);
 1856   }
 1857 
 1858   /* Will ASSUME curl_global_init has been called by this point.
 1859      It is not thread-safe to call it here. */
 1860   curl = curl_easy_init();
 1861    
 1862   res = curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, verifyPeer);
 1863   if (res != CURLE_OK)
 1864   {
 1865      ffpmsg("ERROR: CFITSIO was built with a libcurl library that ");
 1866      ffpmsg("does not have SSL support, and therefore can't perform https or ftps transfers.");
 1867      return (FILE_NOT_OPENED);    
 1868   }
 1869   curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, verifyHost);
 1870   
 1871   curl_easy_setopt(curl, CURLOPT_VERBOSE, (long)curl_verbose);
 1872   curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curlToMemCallback);
 1873   snprintf(agentStr,MAXLEN,"User-Agent: FITSIO/HEASARC/%-8.3f",ffvers(&version)); 
 1874   curl_easy_setopt(curl, CURLOPT_USERAGENT,agentStr);
 1875   
 1876   buffer->memory = 0; /* malloc/realloc will grow this in the callback function */
 1877   buffer->size = 0;
 1878   curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)buffer);
 1879   curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curlErrBuf);
 1880   curlErrBuf[0]=0;
 1881   /* This is needed for easy_perform to return an error whenever http server
 1882       returns an error >= 400, ie. if it can't find the requested file. */
 1883   curl_easy_setopt(curl, CURLOPT_FAILONERROR,  1L);
 1884   /* This turns on automatic decompression for all recognized types. */
 1885   curl_easy_setopt(curl, CURLOPT_ENCODING, "");
 1886   
 1887   /* tmpUrl should be large enough to accomodate original url + ".gz" */
 1888   tmpUrl = (char *)malloc(strlen(url)+4);
 1889   strcpy(tmpUrl, url);
 1890   if (show_fits_download_progress)
 1891   {
 1892      curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, curlProgressCallback);
 1893      curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, tmpUrl);
 1894      curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
 1895   }
 1896   else
 1897      curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L);
 1898   
 1899   /* USESSL only necessary for ftps, though it may not hurt anything
 1900      if it were also set for https. */
 1901   if (isFtp)
 1902   {
 1903      curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_ALL);
 1904      if (username)
 1905         curl_easy_setopt(curl, CURLOPT_USERNAME, username);
 1906      if (password)
 1907         curl_easy_setopt(curl, CURLOPT_PASSWORD, password);
 1908   }
 1909   
 1910   /* Unless url already contains a .gz, .Z or '?' (probably from a cgi script),
 1911      first try with .gz appended. */
 1912   
 1913   if (experimentWithCompression)
 1914      strcat(tmpUrl, ".gz");
 1915 
 1916   /* First attempt: verification on */
 1917   curl_easy_setopt(curl, CURLOPT_URL, tmpUrl);
 1918   res = curl_easy_perform(curl);
 1919   if (res != CURLE_OK && res != CURLE_HTTP_RETURNED_ERROR && 
 1920                 res != CURLE_REMOTE_FILE_NOT_FOUND)
 1921   {
 1922      /*   CURLE_HTTP_RETURNED_ERROR is what gets returned if HTTP server
 1923         returns an error code >= 400. CURLE_REMOTE_FILE_NOT_FOUND may
 1924         be returned by an ftp server. If these are not causing this error, 
 1925         assume it is a verification issue. 
 1926           Try again with verification removed, unless user disallowed it
 1927         via environment variable. */
 1928      verify = getenv("CFITSIO_VERIFY_HTTPS");
 1929      if (verify)
 1930      {
 1931         if (verify[0] == 'T' || verify[0] == 't')
 1932         {
 1933            snprintf(errStr,MAXLEN,"libcurl error: %d",res);
 1934            ffpmsg(errStr);
 1935            if (strlen(curlErrBuf))
 1936               ffpmsg(curlErrBuf);     
 1937            curl_easy_cleanup(curl);  
 1938            free(tmpUrl);
 1939            return (FILE_NOT_OPENED);
 1940         }
 1941      }
 1942      verifyPeer = 0;
 1943      verifyHost = 0;
 1944      curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, verifyPeer);
 1945      curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, verifyHost);
 1946      /* Second attempt: no verification, .gz appended */
 1947      res = curl_easy_perform(curl);
 1948      if (res != CURLE_OK)
 1949      {
 1950         if (isFtp && experimentWithCompression)
 1951         {
 1952            strcpy(tmpUrl, url);
 1953            strcat(tmpUrl, ".Z");
 1954            curl_easy_setopt(curl, CURLOPT_URL, tmpUrl);
 1955            /* For ftps, make another attempt with .Z */
 1956            res = curl_easy_perform(curl);
 1957            if (res == CURLE_OK)
 1958            {
 1959               /* Success, but should still warn */
 1960               fprintf(stderr, "Warning: Unable to perform SSL verification on https transfer from: %s\n",
 1961                    tmpUrl);
 1962               notFound=0;          
 1963            }
 1964         }
 1965           
 1966         /* If we've been appending .gz or .Z, try a final time without. */
 1967         if (experimentWithCompression && notFound)
 1968         {
 1969            strcpy(tmpUrl, url);
 1970            curl_easy_setopt(curl, CURLOPT_URL, tmpUrl);
 1971            /* attempt with no verification, no .gz or .Z appended */ 
 1972            res = curl_easy_perform(curl);
 1973            if (res != CURLE_OK)
 1974            {
 1975               snprintf(errStr,MAXLEN,"libcurl error: %d",res);
 1976               ffpmsg(errStr);
 1977               if (strlen(curlErrBuf))
 1978                  ffpmsg(curlErrBuf);     
 1979               curl_easy_cleanup(curl);  
 1980               free(tmpUrl);
 1981               return (FILE_NOT_OPENED);
 1982            }
 1983            else
 1984               /* Success, but should still warn */
 1985               fprintf(stderr, "Warning: Unable to perform SSL verification on https transfer from: %s\n",
 1986                    tmpUrl);           
 1987         }
 1988         else if (notFound)
 1989         {
 1990            snprintf(errStr,MAXLEN,"libcurl error: %d",res);
 1991            ffpmsg(errStr);
 1992            if (strlen(curlErrBuf))
 1993               ffpmsg(curlErrBuf);     
 1994            curl_easy_cleanup(curl);  
 1995            free(tmpUrl);
 1996            return (FILE_NOT_OPENED);
 1997         }        
 1998      }
 1999      else
 2000         /* Success, but still issue warning */
 2001         fprintf(stderr, "Warning: Unable to perform SSL verification on https transfer from: %s\n",
 2002              tmpUrl);
 2003 
 2004   }
 2005   else if (res == CURLE_HTTP_RETURNED_ERROR || res == CURLE_REMOTE_FILE_NOT_FOUND)
 2006   {
 2007      /* .gz extension failed and verification isn't the problem.  
 2008          No need to relax peer/host checking */
 2009      /* Unless url already contained a .gz, .Z or '?' (probably from a cgi script),
 2010         try again with original url unappended (but first try .Z if this is ftps). */
 2011      if (experimentWithCompression)
 2012      {
 2013         if (isFtp)
 2014         {
 2015            strcpy(tmpUrl, url);
 2016            strcat(tmpUrl, ".Z");
 2017            curl_easy_setopt(curl, CURLOPT_URL, tmpUrl); 
 2018            res = curl_easy_perform(curl);
 2019            if (res == CURLE_OK)
 2020               notFound = 0;
 2021         }
 2022         if (notFound)
 2023         {
 2024            strcpy(tmpUrl, url);
 2025            curl_easy_setopt(curl, CURLOPT_URL, tmpUrl); 
 2026            res = curl_easy_perform(curl);
 2027            if (res != CURLE_OK)
 2028            {
 2029               snprintf(errStr,MAXLEN,"libcurl error: %d",res);
 2030               ffpmsg(errStr);
 2031               if (strlen(curlErrBuf))
 2032                  ffpmsg(curlErrBuf);     
 2033               curl_easy_cleanup(curl);  
 2034               free(tmpUrl);
 2035               return (FILE_NOT_OPENED);
 2036            }
 2037         }
 2038      }
 2039      else
 2040      {
 2041         snprintf(errStr,MAXLEN,"libcurl error: %d",res);
 2042         ffpmsg(errStr);
 2043         if (strlen(curlErrBuf))
 2044            ffpmsg(curlErrBuf);     
 2045         curl_easy_cleanup(curl);  
 2046         free(tmpUrl);
 2047         return (FILE_NOT_OPENED);
 2048      }
 2049   }
 2050   
 2051   /* If we made it here, assume tmpUrl was successful. Calling routines
 2052      must make sure url can hold up to 3 extra chars */
 2053   strcpy(url, tmpUrl);
 2054   
 2055   free(tmpUrl);
 2056   curl_easy_cleanup(curl);
 2057   
 2058   #else
 2059    ffpmsg("ERROR: This CFITSIO build was not compiled with the libcurl library package ");
 2060    ffpmsg("and therefore it cannot perform HTTPS or FTPS connections."); 
 2061    return (FILE_NOT_OPENED);  
 2062   
 2063   #endif
 2064   return 0;
 2065 }
 2066 
 2067 /*--------------------------------------------------------------------------*/
 2068 /* This creates a memory file handle with a copy of the URL in filename. The 
 2069    file is uncompressed if necessary */
 2070 
 2071 int ftp_open(char *filename, int rwmode, int *handle)
 2072 {
 2073   FILE *ftpfile;
 2074   FILE *command;
 2075   int sock;
 2076   char errorstr[MAXLEN];
 2077   char recbuf[MAXLEN];
 2078   long len;
 2079   int status;
 2080   char firstchar;
 2081 
 2082   closememfile = 0;
 2083   closecommandfile = 0;
 2084   closeftpfile = 0;
 2085 
 2086   /* don't do r/w files */
 2087   if (rwmode != 0) {
 2088     ffpmsg("Can't open ftp:// type file with READWRITE access");
 2089     ffpmsg("Specify an outfile for r/w access (ftp_open)");
 2090     return (FILE_NOT_OPENED);
 2091   }
 2092 
 2093   /* do the signal handler bits */
 2094   if (setjmp(env) != 0) {
 2095     /* feels like the second time */
 2096     /* this means something bad happened */
 2097     ffpmsg("Timeout (ftp_open)");
 2098     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 2099     ffpmsg(errorstr);
 2100     ffpmsg("   (multiplied x10 for files requiring uncompression)");
 2101     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 2102     goto error;
 2103   }
 2104 
 2105   signal(SIGALRM, signal_handler);
 2106   
 2107   /* Open the ftp connetion.  ftpfile is connected to the file port, 
 2108      command is connected to port 21.  sock is the socket on port 21 */
 2109 
 2110   if (strlen(filename) > MAXLEN - 4) {
 2111       ffpmsg("filename too long (ftp_open)");
 2112       ffpmsg(filename);
 2113       goto error;
 2114   } 
 2115 
 2116   alarm(net_timeout);
 2117   if (ftp_open_network(filename,&ftpfile,&command,&sock)) {
 2118 
 2119       alarm(0);
 2120       ffpmsg("Unable to open following ftp file (ftp_open):");
 2121       ffpmsg(filename);
 2122       goto error;
 2123   } 
 2124 
 2125   closeftpfile++;
 2126   closecommandfile++;
 2127 
 2128   /* create the memory file */
 2129   if ((status = mem_create(filename,handle))) {
 2130     ffpmsg ("Could not create memory file to passive port (ftp_open)");
 2131     ffpmsg(filename);
 2132     goto error;
 2133   }
 2134   closememfile++;
 2135   /* This isn't quite right, it'll fail if the file has .gzabc at the end
 2136      for instance */
 2137 
 2138   /* Decide if the file is compressed */
 2139   firstchar = fgetc(ftpfile);
 2140   ungetc(firstchar,ftpfile);
 2141 
 2142   if (strstr(filename,".gz") || 
 2143       strstr(filename,".Z") ||
 2144       ('\037' == firstchar)) {
 2145     
 2146     status = 0;
 2147     /* A bit arbritary really, the user will probably hit ^C */
 2148     alarm(net_timeout*10);
 2149     status = mem_uncompress2mem(filename, ftpfile, *handle);
 2150     alarm(0);
 2151     if (status) {
 2152       ffpmsg("Error writing compressed memory file (ftp_open)");
 2153       ffpmsg(filename);
 2154       goto error;
 2155     }
 2156   } else {
 2157     /* write a memory file */
 2158     alarm(net_timeout);
 2159     while(0 != (len = fread(recbuf,1,MAXLEN,ftpfile))) {
 2160       alarm(0);
 2161       status = mem_write(*handle,recbuf,len);
 2162       if (status) {
 2163     ffpmsg("Error writing memory file (http_open)");
 2164         ffpmsg(filename);
 2165     goto error;
 2166       }
 2167       alarm(net_timeout);
 2168     }
 2169   }
 2170 
 2171   /* close and clean up */
 2172   fclose(ftpfile);
 2173   closeftpfile--;
 2174 
 2175   fclose(command);
 2176   NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2177   closecommandfile--;
 2178 
 2179   signal(SIGALRM, SIG_DFL);
 2180   alarm(0);
 2181 
 2182   return mem_seek(*handle,0);
 2183 
 2184  error:
 2185   alarm(0); /* clear it */
 2186   if (closecommandfile) {
 2187     fclose(command);
 2188     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2189   }
 2190   if (closeftpfile) {
 2191     fclose(ftpfile);
 2192   }
 2193   if (closememfile) {
 2194     mem_close_free(*handle);
 2195   }
 2196   
 2197   signal(SIGALRM, SIG_DFL);
 2198   return (FILE_NOT_OPENED);
 2199 }
 2200 /*--------------------------------------------------------------------------*/
 2201 /* This creates a file handle with a copy of the URL in filename. The 
 2202    file must be  uncompressed and is copied to disk first */
 2203 
 2204 int ftp_file_open(char *url, int rwmode, int *handle)
 2205 {
 2206   FILE *ftpfile;
 2207   FILE *command;
 2208   char errorstr[MAXLEN];
 2209   char recbuf[MAXLEN];
 2210   long len;
 2211   int sock;
 2212   int ii, flen, status;
 2213   char firstchar;
 2214 
 2215   /* Check if output file is actually a memory file */
 2216   if (!strncmp(netoutfile, "mem:", 4) )
 2217   {
 2218      /* allow the memory file to be opened with write access */
 2219      return( ftp_open(url, READONLY, handle) );
 2220   }     
 2221 
 2222   closeftpfile = 0;
 2223   closecommandfile = 0;
 2224   closefile = 0;
 2225   closeoutfile = 0;
 2226   
 2227   /* cfileio made a mistake, need to know where to write the output file */
 2228   flen = strlen(netoutfile);
 2229   if (!flen) 
 2230     {
 2231       ffpmsg("Output file not set, shouldn't have happened (ftp_file_open)");
 2232       return (FILE_NOT_OPENED);
 2233     }
 2234 
 2235   /* do the signal handler bits */
 2236   if (setjmp(env) != 0) {
 2237     /* feels like the second time */
 2238     /* this means something bad happened */
 2239     ffpmsg("Timeout (ftp_file_open)");
 2240     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 2241     ffpmsg(errorstr);
 2242     ffpmsg("   (multiplied x10 for files requiring uncompression)");
 2243     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 2244     goto error;
 2245   }
 2246 
 2247   signal(SIGALRM, signal_handler);
 2248   
 2249   /* open the network connection to url. ftpfile holds the connection to
 2250      the input file, command holds the connection to port 21, and sock is 
 2251      the socket connected to port 21 */
 2252 
 2253   alarm(net_timeout);
 2254   if ((status = ftp_open_network(url,&ftpfile,&command,&sock))) {
 2255     alarm(0);
 2256     ffpmsg("Unable to open http file (ftp_file_open)");
 2257     ffpmsg(url);
 2258     goto error;
 2259   }
 2260   closeftpfile++;
 2261   closecommandfile++;
 2262 
 2263   if (*netoutfile == '!')
 2264   {
 2265      /* user wants to clobber file, if it already exists */
 2266      for (ii = 0; ii < flen; ii++)
 2267          netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
 2268 
 2269      status = file_remove(netoutfile);
 2270   }
 2271 
 2272   /* Now, what do we do with the file */
 2273   firstchar = fgetc(ftpfile);
 2274   ungetc(firstchar,ftpfile);
 2275 
 2276   if (strstr(url,".gz") || 
 2277       strstr(url,".Z") ||
 2278       ('\037' == firstchar)) {
 2279 
 2280     /* to make this more cfitsioish we use the file driver calls to create
 2281        the file */
 2282     /* Create the output file */
 2283     if ((status =  file_create(netoutfile,handle))) {
 2284       ffpmsg("Unable to create output file (ftp_file_open)");
 2285       ffpmsg(netoutfile);
 2286       goto error;
 2287     }
 2288 
 2289     file_close(*handle);
 2290     if (NULL == (outfile = fopen(netoutfile,"w"))) {
 2291       ffpmsg("Unable to reopen the output file (ftp_file_open)");
 2292       ffpmsg(netoutfile);
 2293       goto error;
 2294     }
 2295     closeoutfile++;
 2296     status = 0;
 2297 
 2298     /* Ok, this is a tough case, let's be arbritary and say 10*net_timeout,
 2299        Given the choices for nettimeout above they'll probaby ^C before, but
 2300        it's always worth a shot*/
 2301 
 2302     alarm(net_timeout*10);
 2303     status = uncompress2file(url,ftpfile,outfile,&status);
 2304     alarm(0);
 2305     if (status) {
 2306       ffpmsg("Unable to uncompress the output file (ftp_file_open)");
 2307       ffpmsg(url);
 2308       ffpmsg(netoutfile);
 2309       goto error;
 2310     }
 2311     fclose(outfile);
 2312     closeoutfile--;
 2313 
 2314   } else {
 2315     
 2316     /* Create the output file */
 2317     if ((status =  file_create(netoutfile,handle))) {
 2318       ffpmsg("Unable to create output file (ftp_file_open)");
 2319       ffpmsg(netoutfile);
 2320       goto error;
 2321     }
 2322     closefile++;
 2323     
 2324     /* write a file */
 2325     alarm(net_timeout);
 2326     while(0 != (len = fread(recbuf,1,MAXLEN,ftpfile))) {
 2327       alarm(0);
 2328       status = file_write(*handle,recbuf,len);
 2329       if (status) {
 2330     ffpmsg("Error writing file (ftp_file_open)");
 2331         ffpmsg(url);
 2332         ffpmsg(netoutfile);
 2333     goto error;
 2334       }
 2335       alarm(net_timeout);
 2336     }
 2337     file_close(*handle);
 2338   }
 2339   fclose(ftpfile);
 2340   closeftpfile--;
 2341   
 2342   fclose(command);
 2343   NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2344   closecommandfile--;
 2345 
 2346   signal(SIGALRM, SIG_DFL);
 2347   alarm(0);
 2348 
 2349   return file_open(netoutfile,rwmode,handle);
 2350 
 2351  error:
 2352   alarm(0); /* clear it */
 2353   if (closeftpfile) {
 2354     fclose(ftpfile);
 2355   }
 2356   if (closecommandfile) {
 2357     fclose(command);
 2358     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2359   }
 2360   if (closeoutfile) {
 2361     fclose(outfile);
 2362   }
 2363   if (closefile) {
 2364     file_close(*handle);
 2365   } 
 2366   
 2367   signal(SIGALRM, SIG_DFL);
 2368   return (FILE_NOT_OPENED);
 2369 }
 2370 
 2371 /*--------------------------------------------------------------------------*/
 2372 /* This creates a memory  handle with a copy of the URL in filename. The 
 2373    file must be compressed and is copied to disk first */
 2374 
 2375 int ftp_compress_open(char *url, int rwmode, int *handle)
 2376 {
 2377   FILE *ftpfile;
 2378   FILE *command;
 2379   char errorstr[MAXLEN];
 2380   char recbuf[MAXLEN];
 2381   long len;
 2382   int ii, flen, status;
 2383   int sock;
 2384   char firstchar;
 2385 
 2386   closeftpfile = 0;
 2387   closecommandfile = 0;
 2388   closememfile = 0;
 2389   closefdiskfile = 0;
 2390   closediskfile = 0;
 2391 
 2392   /* don't do r/w files */
 2393   if (rwmode != 0) {
 2394     ffpmsg("Compressed files must be r/o");
 2395     return (FILE_NOT_OPENED);
 2396   }
 2397   
 2398   /* Need to know where to write the output file */
 2399   flen = strlen(netoutfile);
 2400   if (!flen) 
 2401     {
 2402       ffpmsg(
 2403     "Output file not set, shouldn't have happened (ftp_compress_open)");
 2404       return (FILE_NOT_OPENED);
 2405     }
 2406   
 2407   /* do the signal handler bits */
 2408   if (setjmp(env) != 0) {
 2409     /* feels like the second time */
 2410     /* this means something bad happened */
 2411     ffpmsg("Timeout (ftp_compress_open)");
 2412     snprintf(errorstr, MAXLEN, "Download timeout exceeded: %d seconds",net_timeout);
 2413     ffpmsg(errorstr);
 2414     ffpmsg("   Timeout may be adjusted with fits_set_timeout");
 2415     goto error;
 2416   }
 2417   
 2418   signal(SIGALRM, signal_handler);
 2419   
 2420   /* Open the network connection to url, ftpfile is connected to the file 
 2421      port, command is connected to port 21.  sock is for writing to port 21 */
 2422   alarm(net_timeout);
 2423 
 2424   if ((status = ftp_open_network(url,&ftpfile,&command,&sock))) {
 2425     alarm(0);
 2426     ffpmsg("Unable to open ftp file (ftp_compress_open)");
 2427     ffpmsg(url);
 2428     goto error;
 2429   }
 2430   closeftpfile++;
 2431   closecommandfile++;
 2432 
 2433   /* Now, what do we do with the file */
 2434   firstchar = fgetc(ftpfile);
 2435   ungetc(firstchar,ftpfile);
 2436 
 2437   if (strstr(url,".gz") || 
 2438       strstr(url,".Z") ||
 2439       ('\037' == firstchar)) {
 2440   
 2441     if (*netoutfile == '!')
 2442     {
 2443        /* user wants to clobber file, if it already exists */
 2444        for (ii = 0; ii < flen; ii++)
 2445           netoutfile[ii] = netoutfile[ii + 1];  /* remove '!' */
 2446 
 2447        status = file_remove(netoutfile);
 2448     }
 2449 
 2450     /* Create the output file */
 2451     if ((status =  file_create(netoutfile,handle))) {
 2452       ffpmsg("Unable to create output file (ftp_compress_open)");
 2453       ffpmsg(netoutfile);
 2454       goto error;
 2455     }
 2456     closediskfile++;
 2457     
 2458     /* write a file */
 2459     alarm(net_timeout);
 2460     while(0 != (len = fread(recbuf,1,MAXLEN,ftpfile))) {
 2461       alarm(0);
 2462       status = file_write(*handle,recbuf,len);
 2463       if (status) {
 2464     ffpmsg("Error writing file (ftp_compres_open)");
 2465         ffpmsg(url);
 2466         ffpmsg(netoutfile);
 2467     goto error;
 2468       }
 2469       alarm(net_timeout);
 2470     }
 2471 
 2472     file_close(*handle);
 2473     closediskfile--;
 2474     fclose(ftpfile);
 2475     closeftpfile--;
 2476     /* Close down the ftp connection */
 2477     fclose(command);
 2478     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2479     closecommandfile--;
 2480 
 2481     /* File is on disk, let's uncompress it into memory */
 2482 
 2483     if (NULL == (diskfile = fopen(netoutfile,"r"))) {
 2484       ffpmsg("Unable to reopen disk file (ftp_compress_open)");
 2485       ffpmsg(netoutfile);
 2486       return (FILE_NOT_OPENED);
 2487     }
 2488     closefdiskfile++;
 2489   
 2490     if ((status =  mem_create(url,handle))) {
 2491       ffpmsg("Unable to create memory file (ftp_compress_open)");
 2492       ffpmsg(url);
 2493       goto error;
 2494     }
 2495     closememfile++;
 2496 
 2497     status = 0;
 2498     status = mem_uncompress2mem(url,diskfile,*handle);
 2499     fclose(diskfile);
 2500     closefdiskfile--;
 2501 
 2502     if (status) {
 2503       ffpmsg("Error writing compressed memory file (ftp_compress_open)");
 2504       goto error;
 2505     }
 2506       
 2507   } else {
 2508     /* Opps, this should not have happened */
 2509     ffpmsg("Can only compressed files here (ftp_compress_open)");
 2510     goto error;
 2511   }    
 2512     
 2513 
 2514   signal(SIGALRM, SIG_DFL);
 2515   alarm(0);
 2516   return mem_seek(*handle,0);
 2517 
 2518  error:
 2519   alarm(0); /* clear it */
 2520   if (closeftpfile) {
 2521     fclose(ftpfile);
 2522   }
 2523   if (closecommandfile) {
 2524     fclose(command);
 2525     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2526   }
 2527   if (closefdiskfile) {
 2528     fclose(diskfile);
 2529   }
 2530   if (closememfile) {
 2531     mem_close_free(*handle);
 2532   }
 2533   if (closediskfile) {
 2534     file_close(*handle);
 2535   } 
 2536   
 2537   signal(SIGALRM, SIG_DFL);
 2538   return (FILE_NOT_OPENED);
 2539 }
 2540 
 2541 /*--------------------------------------------------------------------------*/
 2542 /* Open a ftp connection to filename (really a URL), return ftpfile set to 
 2543    the file connection, and command set to the control connection, with sock
 2544    also set to the control connection */
 2545 
 2546 static int ftp_open_network(char *filename, FILE **ftpfile, FILE **command, int *sock)
 2547 {
 2548   int status;
 2549   int sock1;
 2550   int tmpint;
 2551   char recbuf[MAXLEN];
 2552   char errorstr[MAXLEN];
 2553   char tmpstr[MAXLEN];
 2554   char proto[SHORTLEN];
 2555   char host[SHORTLEN];
 2556   char agentStr[SHORTLEN];
 2557   char *newhost;
 2558   char *username;
 2559   char *password;
 2560   char fn[MAXLEN];
 2561   char *newfn;
 2562   char *passive;
 2563   char *tstr;
 2564   char *saveptr;
 2565   char ip[SHORTLEN];
 2566   char turl[MAXLEN];
 2567   int port;
 2568   int ii,tryingtologin = 1;
 2569   float version=0.0;
 2570 
 2571   /* parse the URL */
 2572   if (strlen(filename) > MAXLEN - 7) {
 2573     ffpmsg("ftp filename is too long (ftp_open_network)");
 2574     return (FILE_NOT_OPENED);
 2575   }
 2576 
 2577   strcpy(turl,"ftp://");
 2578   strcat(turl,filename);
 2579   if (NET_ParseUrl(turl,proto,host,&port,fn)) {
 2580     snprintf(errorstr,MAXLEN,"URL Parse Error (ftp_open) %s",filename);
 2581     ffpmsg(errorstr);
 2582     return (FILE_NOT_OPENED);
 2583   }
 2584   
 2585   port = 21;
 2586   /* We might have a user name.  If not, set defaults for username and password */
 2587   username = "anonymous";
 2588   snprintf(agentStr,SHORTLEN,"User-Agent: FITSIO/HEASARC/%-8.3f",ffvers(&version));
 2589   password = agentStr;
 2590   /* is there an @ sign */
 2591   if (NULL != (newhost = strrchr(host,'@'))) {
 2592     *newhost = '\0'; /* make it a null, */
 2593     newhost++; /* Now newhost points to the host name and host points to the 
 2594           user name, password combo */
 2595     username = host;
 2596     /* is there a : for a password */
 2597     if (NULL != strchr(username,':')) {
 2598       password = strchr(username,':');
 2599       *password = '\0';
 2600       password++;
 2601     }
 2602   } else {
 2603     newhost = host;
 2604   }
 2605 
 2606   for (ii = 0; ii < 10; ii++) {  /* make up to 10 attempts to log in */
 2607   
 2608     /* Connect to the host on the required port */
 2609     *sock = NET_TcpConnect(newhost,port);
 2610     /* convert it to a stdio file */
 2611     if (NULL == (*command = fdopen(*sock,"r"))) {
 2612       ffpmsg ("fdopen failed to convert socket to stdio file (ftp_open_netowrk)");
 2613       return (FILE_NOT_OPENED);
 2614     }
 2615 
 2616     /* Wait for the 220 response */
 2617     if (ftp_status(*command,"220 ")) {
 2618       fclose(*command);
 2619       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2620 
 2621 /*      ffpmsg("sleeping for 5 in ftp_open_network, then try again"); */
 2622 
 2623       sleep (5);  /* take a nap and hope ftp server sorts itself out in the meantime */
 2624 
 2625     } else {
 2626       tryingtologin = 0;
 2627       break;
 2628     }
 2629   }
 2630 
 2631   if (tryingtologin) { /* the 10 attempts were not successful */
 2632      ffpmsg ("error connecting to remote server, no 220 seen (ftp_open_network)");
 2633      return (FILE_NOT_OPENED);
 2634   }
 2635 
 2636   /* Send the user name and wait for the right response */
 2637   snprintf(tmpstr,MAXLEN,"USER %s\r\n",username);
 2638 
 2639   status = NET_SendRaw(*sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2640 
 2641   if (ftp_status(*command,"331 ")) {
 2642     ffpmsg ("USER error no 331 seen (ftp_open_network)");
 2643     fclose(*command);
 2644     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2645     return (FILE_NOT_OPENED);
 2646   }
 2647   
 2648   /* Send the password and wait for the right response */
 2649   snprintf(tmpstr,MAXLEN,"PASS %s\r\n",password);
 2650   status = NET_SendRaw(*sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2651   
 2652   if (ftp_status(*command,"230 ")) {
 2653     ffpmsg ("PASS error, no 230 seen (ftp_open_network)");
 2654     fclose(*command);
 2655     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2656     return (FILE_NOT_OPENED);
 2657   }
 2658 
 2659   /* now do the cwd command */
 2660   newfn = strrchr(fn,'/');
 2661   if (newfn == NULL) {
 2662     strcpy(tmpstr,"CWD /\r\n");
 2663     newfn = fn;
 2664   } else {
 2665     *newfn = '\0';
 2666     newfn++;
 2667     if (strlen(fn) == 0) {
 2668       strcpy(tmpstr,"CWD /\r\n");
 2669     } else {
 2670       /* remove the leading slash */
 2671       if (fn[0] == '/') {
 2672     snprintf(tmpstr,MAXLEN,"CWD %s\r\n",&fn[1]);
 2673       } else {
 2674     snprintf(tmpstr,MAXLEN,"CWD %s\r\n",fn);
 2675       } 
 2676     }
 2677   }
 2678   
 2679   status = NET_SendRaw(*sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2680   
 2681   if (ftp_status(*command,"250 ")) {
 2682     ffpmsg ("CWD error, no 250 seen (ftp_open_network)");
 2683     fclose(*command);
 2684     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2685     return (FILE_NOT_OPENED);
 2686   }
 2687   
 2688   if (!strlen(newfn)) {
 2689     ffpmsg("Null file name (ftp_open)");
 2690     fclose(*command);
 2691     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2692     return (FILE_NOT_OPENED);
 2693   }
 2694 
 2695   /* Always use binary mode */
 2696   snprintf(tmpstr,MAXLEN,"TYPE I\r\n");
 2697   status = NET_SendRaw(*sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2698   
 2699   if (ftp_status(*command,"200 ")) {
 2700     ffpmsg ("TYPE I error, 200 not seen (ftp_open_network)");
 2701     fclose(*command);
 2702     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2703     return (FILE_NOT_OPENED);
 2704   }
 2705  
 2706   status = NET_SendRaw(*sock,"PASV\r\n",6,NET_DEFAULT);
 2707 
 2708   if (!(fgets(recbuf,MAXLEN,*command))) {
 2709     ffpmsg ("PASV error (ftp_open)");
 2710     fclose(*command);
 2711     NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2712     return (FILE_NOT_OPENED);
 2713   }
 2714   
 2715   /*  Passive mode response looks like
 2716       227 Entering Passive Mode (129,194,67,8,210,80) */
 2717   if (recbuf[0] == '2' && recbuf[1] == '2' && recbuf[2] == '7') {
 2718     /* got a good passive mode response, find the opening ( */
 2719     
 2720     if (!(passive = strchr(recbuf,'('))) {
 2721       ffpmsg ("PASV error (ftp_open_network)");
 2722       fclose(*command);
 2723       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2724       return (FILE_NOT_OPENED);
 2725     }
 2726     
 2727     *passive = '\0';
 2728     passive++;
 2729     ip[0] = '\0';
 2730       
 2731     /* Messy parsing of response from PASV *command */
 2732     
 2733     if (!(tstr = ffstrtok(passive,",)",&saveptr))) {
 2734       ffpmsg ("PASV error (ftp_open_network)");
 2735       fclose(*command);
 2736       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2737       return (FILE_NOT_OPENED);
 2738     }
 2739     strcpy(ip,tstr);
 2740     strcat(ip,".");
 2741     
 2742     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 2743       ffpmsg ("PASV error (ftp_open_network)");
 2744       fclose(*command);
 2745       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2746       return (FILE_NOT_OPENED);
 2747     }
 2748     strcat(ip,tstr);
 2749     strcat(ip,".");
 2750     
 2751     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 2752       ffpmsg ("PASV error (ftp_open_network)");
 2753       fclose(*command);
 2754       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2755       return (FILE_NOT_OPENED);
 2756     }
 2757     strcat(ip,tstr);
 2758     strcat(ip,".");
 2759     
 2760     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 2761       ffpmsg ("PASV error (ftp_open_network)");
 2762       fclose(*command);
 2763       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2764       return (FILE_NOT_OPENED);
 2765     }
 2766     strcat(ip,tstr);
 2767     
 2768     /* Done the ip number, now do the port # */
 2769     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 2770       ffpmsg ("PASV error (ftp_open_network)");
 2771       fclose(*command);
 2772       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2773       return (FILE_NOT_OPENED);
 2774     }
 2775     sscanf(tstr,"%d",&port);
 2776     port *= 256;
 2777     
 2778     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 2779       ffpmsg ("PASV error (ftp_open_network)");
 2780       fclose(*command);
 2781       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2782       return (FILE_NOT_OPENED);
 2783     }
 2784     sscanf(tstr,"%d",&tmpint);
 2785     port += tmpint;
 2786 
 2787     if (!strlen(newfn)) {
 2788       ffpmsg("Null file name (ftp_open_network)");
 2789       fclose(*command);
 2790       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2791       return (FILE_NOT_OPENED);
 2792     }
 2793     
 2794     /* Connect to the data port */
 2795     sock1 = NET_TcpConnect(ip,port);
 2796     if (NULL == (*ftpfile = fdopen(sock1,"r"))) {
 2797       ffpmsg ("Could not connect to passive port (ftp_open_network)");
 2798       fclose(*command);
 2799       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2800       return (FILE_NOT_OPENED);
 2801     }
 2802 
 2803     /* Send the retrieve command */
 2804     snprintf(tmpstr,MAXLEN,"RETR %s\r\n",newfn);
 2805     status = NET_SendRaw(*sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2806 
 2807     if (ftp_status(*command,"150 ")) {
 2808       fclose(*ftpfile);
 2809       NET_SendRaw(sock1,"QUIT\r\n",6,NET_DEFAULT);
 2810       fclose(*command);
 2811       NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2812       return (FILE_NOT_OPENED);
 2813     }
 2814     return 0;    /* successfully opened the ftp file */
 2815   }
 2816   
 2817   /* no passive mode */
 2818 
 2819   fclose(*command);
 2820   NET_SendRaw(*sock,"QUIT\r\n",6,NET_DEFAULT);
 2821   return (FILE_NOT_OPENED);
 2822 }
 2823 /*--------------------------------------------------------------------------*/
 2824 /* Open a ftp connection to see if the file exists (return 1) or not (return 0) */
 2825 
 2826 int ftp_file_exist(char *filename)
 2827 {
 2828   FILE *ftpfile;
 2829   FILE *command;
 2830   int sock;
 2831   int status;
 2832   int sock1;
 2833   int tmpint;
 2834   char recbuf[MAXLEN];
 2835   char errorstr[MAXLEN];
 2836   char tmpstr[MAXLEN];
 2837   char proto[SHORTLEN];
 2838   char host[SHORTLEN];
 2839   char *newhost;
 2840   char *username;
 2841   char *password;
 2842   char fn[MAXLEN];
 2843   char *newfn;
 2844   char *passive;
 2845   char *tstr;
 2846   char *saveptr;
 2847   char ip[SHORTLEN];
 2848   char turl[MAXLEN];
 2849   int port;
 2850   int ii, tryingtologin = 1;
 2851 
 2852   /* parse the URL */
 2853   if (strlen(filename) > MAXLEN - 7) {
 2854     ffpmsg("ftp filename is too long (ftp_file_exist)");
 2855     return 0;
 2856   }
 2857 
 2858   strcpy(turl,"ftp://");
 2859   strcat(turl,filename);
 2860   if (NET_ParseUrl(turl,proto,host,&port,fn)) {
 2861     snprintf(errorstr,MAXLEN,"URL Parse Error (ftp_file_exist) %s",filename);
 2862     ffpmsg(errorstr);
 2863     return 0;
 2864   }
 2865 
 2866   port = 21;
 2867   /* we might have a user name */
 2868   username = "anonymous";
 2869   password = "user@host.com";
 2870   /* is there an @ sign */
 2871   if (NULL != (newhost = strrchr(host,'@'))) {
 2872     *newhost = '\0'; /* make it a null, */
 2873     newhost++; /* Now newhost points to the host name and host points to the 
 2874           user name, password combo */
 2875     username = host;
 2876     /* is there a : for a password */
 2877     if (NULL != strchr(username,':')) {
 2878       password = strchr(username,':');
 2879       *password = '\0';
 2880       password++;
 2881     }
 2882   } else {
 2883     newhost = host;
 2884   }
 2885 
 2886   for (ii = 0; ii < 10; ii++) {  /* make up to 10 attempts to log in */
 2887   
 2888   /* Connect to the host on the required port */
 2889   sock = NET_TcpConnect(newhost,port);
 2890   /* convert it to a stdio file */
 2891   if (NULL == (command = fdopen(sock,"r"))) {
 2892     ffpmsg ("Failed to convert socket to stdio file (ftp_file_exist)");
 2893     return 0;
 2894   }
 2895 
 2896   /* Wait for the 220 response */
 2897   if (ftp_status(command,"220")) {
 2898     ffpmsg ("error connecting to remote server, no 220 seen (ftp_file_exist)");
 2899     fclose(command);
 2900     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2901 
 2902 /*    ffpmsg("sleeping for 5 in ftp_file_exist, then try again"); */
 2903 
 2904     sleep (5);  /* take a nap and hope ftp server sorts itself out in the meantime */
 2905 
 2906   } else {
 2907     tryingtologin = 0;
 2908     break;
 2909   }
 2910   
 2911   }  
 2912 
 2913   if (tryingtologin) { /* the 10 attempts were not successful */
 2914      ffpmsg ("error connecting to remote server, no 220 seen (ftp_open_network)");
 2915      return (0);
 2916   }
 2917  
 2918   /* Send the user name and wait for the right response */
 2919   snprintf(tmpstr,MAXLEN,"USER %s\r\n",username);
 2920 
 2921   status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2922   
 2923   /* If command is refused due to the connection requiring SSL (ie. an
 2924      fpts connection), this is where it will first be detected by way
 2925      of a 550 error code. */
 2926      
 2927   status = ftp_status(command,"331 ");
 2928   if (status == 550)
 2929   {
 2930     ffpmsg ("Server is requesting SSL, will switch to ftps (ftp_file_exist)");
 2931     fclose(command);
 2932     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2933     return -1;
 2934   }
 2935   else if (status) {
 2936     ffpmsg ("USER error no 331 seen (ftp_file_exist)");
 2937     fclose(command);
 2938     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2939     return 0;
 2940   }
 2941   
 2942   /* Send the password and wait for the right response */
 2943   snprintf(tmpstr,MAXLEN,"PASS %s\r\n",password);
 2944   status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2945   
 2946   if (ftp_status(command,"230 ")) {
 2947     ffpmsg ("PASS error, no 230 seen (ftp_file_exist)");
 2948     fclose(command);
 2949     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2950     return 0;
 2951   }
 2952 
 2953   /* now do the cwd command */
 2954   newfn = strrchr(fn,'/');
 2955   if (newfn == NULL) {
 2956     strcpy(tmpstr,"CWD /\r\n");
 2957     newfn = fn;
 2958   } else {
 2959     *newfn = '\0';
 2960     newfn++;
 2961     if (strlen(fn) == 0) {
 2962       strcpy(tmpstr,"CWD /\r\n");
 2963     } else {
 2964       /* remove the leading slash */
 2965       if (fn[0] == '/') {
 2966     snprintf(tmpstr,MAXLEN,"CWD %s\r\n",&fn[1]);
 2967       } else {
 2968     snprintf(tmpstr,MAXLEN,"CWD %s\r\n",fn);
 2969       } 
 2970     }
 2971   }
 2972 
 2973   status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2974   
 2975   if (ftp_status(command,"250 ")) {
 2976     ffpmsg ("CWD error, no 250 seen (ftp_file_exist)");
 2977     fclose(command);
 2978     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2979     return 0;
 2980   }
 2981   
 2982   if (!strlen(newfn)) {
 2983     ffpmsg("Null file name (ftp_file_exist)");
 2984     fclose(command);
 2985     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2986     return 0;
 2987   }
 2988 
 2989   /* Always use binary mode */
 2990   snprintf(tmpstr,MAXLEN,"TYPE I\r\n");
 2991   status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 2992   
 2993   if (ftp_status(command,"200 ")) {
 2994     ffpmsg ("TYPE I error, 200 not seen (ftp_file_exist)");
 2995     fclose(command);
 2996     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 2997     return 0;
 2998   }
 2999 
 3000   status = NET_SendRaw(sock,"PASV\r\n",6,NET_DEFAULT);
 3001 
 3002   if (!(fgets(recbuf,MAXLEN,command))) {
 3003     ffpmsg ("PASV error (ftp_file_exist)");
 3004     fclose(command);
 3005     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3006     return 0;
 3007   }
 3008   
 3009   /*  Passive mode response looks like
 3010       227 Entering Passive Mode (129,194,67,8,210,80) */
 3011   if (recbuf[0] == '2' && recbuf[1] == '2' && recbuf[2] == '7') {
 3012     /* got a good passive mode response, find the opening ( */
 3013     
 3014     if (!(passive = strchr(recbuf,'('))) {
 3015       ffpmsg ("PASV error (ftp_file_exist)");
 3016       fclose(command);
 3017       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3018       return 0;
 3019     }
 3020     
 3021     *passive = '\0';
 3022     passive++;
 3023     ip[0] = '\0';
 3024       
 3025     /* Messy parsing of response from PASV command */
 3026     
 3027     if (!(tstr = ffstrtok(passive,",)",&saveptr))) {
 3028       ffpmsg ("PASV error (ftp_file_exist)");
 3029       fclose(command);
 3030       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3031       return 0;
 3032     }
 3033     strcpy(ip,tstr);
 3034     strcat(ip,".");
 3035     
 3036     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 3037       ffpmsg ("PASV error (ftp_file_exist)");
 3038       fclose(command);
 3039       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3040       return 0;
 3041     }
 3042     strcat(ip,tstr);
 3043     strcat(ip,".");
 3044     
 3045     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 3046       ffpmsg ("PASV error (ftp_file_exist)");
 3047       fclose(command);
 3048       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3049       return 0;
 3050     }
 3051     strcat(ip,tstr);
 3052     strcat(ip,".");
 3053     
 3054     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 3055       ffpmsg ("PASV error (ftp_file_exist)");
 3056       fclose(command);
 3057       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3058       return 0;
 3059     }
 3060     strcat(ip,tstr);
 3061     
 3062     /* Done the ip number, now do the port # */
 3063     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 3064       ffpmsg ("PASV error (ftp_file_exist)");
 3065       fclose(command);
 3066       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3067       return 0;
 3068     }
 3069     sscanf(tstr,"%d",&port);
 3070     port *= 256;
 3071     
 3072     if (!(tstr = ffstrtok(NULL,",)",&saveptr))) {
 3073       ffpmsg ("PASV error (ftp_file_exist)");
 3074       fclose(command);
 3075       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3076       return 0;
 3077     }
 3078     sscanf(tstr,"%d",&tmpint);
 3079     port += tmpint;
 3080 
 3081     if (!strlen(newfn)) {
 3082       ffpmsg("Null file name (ftp_file_exist)");
 3083       fclose(command);
 3084       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3085       return 0;
 3086     }
 3087 
 3088     /* Connect to the data port */
 3089     sock1 = NET_TcpConnect(ip,port);
 3090     if (NULL == (ftpfile = fdopen(sock1,"r"))) {
 3091       ffpmsg ("Could not connect to passive port (ftp_file_exist)");
 3092       fclose(command);
 3093       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3094       return 0;
 3095     }
 3096 
 3097     /* Send the retrieve command */
 3098     snprintf(tmpstr,MAXLEN,"RETR %s\r\n",newfn);
 3099     status = NET_SendRaw(sock,tmpstr,strlen(tmpstr),NET_DEFAULT);
 3100 
 3101     if (ftp_status(command,"150 ")) {
 3102       fclose(ftpfile); 
 3103       NET_SendRaw(sock1,"QUIT\r\n",6,NET_DEFAULT);
 3104       fclose(command);
 3105       NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3106       return 0;
 3107     }
 3108     
 3109     /* if we got here then the file probably exists */
 3110 
 3111     fclose(ftpfile); 
 3112     NET_SendRaw(sock1,"QUIT\r\n",6,NET_DEFAULT);
 3113     fclose(command); 
 3114     NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3115     return 1;
 3116   }
 3117   
 3118   /* no passive mode */
 3119 
 3120   fclose(command);
 3121   NET_SendRaw(sock,"QUIT\r\n",6,NET_DEFAULT);
 3122   return 0;
 3123 }
 3124 
 3125 /*--------------------------------------------------------------------------*/
 3126 /* return a socket which results from connection to hostname on port port */
 3127 int NET_TcpConnect(char *hostname, int port)
 3128 {
 3129   /* Connect to hostname on port */
 3130  
 3131    struct sockaddr_in sockaddr;
 3132    int sock;
 3133    int stat;
 3134    int val = 1;
 3135  
 3136    CreateSocketAddress(&sockaddr,hostname,port);
 3137    /* Create socket */
 3138    if ((sock = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
 3139      ffpmsg("ERROR: NET_TcpConnect can't create socket");
 3140      return CONNECTION_ERROR;
 3141    }
 3142  
 3143    if ((stat = connect(sock, (struct sockaddr*) &sockaddr, 
 3144                sizeof(sockaddr))) 
 3145        < 0) {
 3146      close(sock);
 3147 /*
 3148      perror("NET_Tcpconnect - Connection error");
 3149      ffpmsg("Can't connect to host, connection error");
 3150 */
 3151      return CONNECTION_ERROR;
 3152    }
 3153    setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, (char *)&val, sizeof(val));
 3154    setsockopt(sock, SOL_SOCKET,  SO_KEEPALIVE, (char *)&val, sizeof(val));
 3155 
 3156    val = 65536;
 3157    setsockopt(sock, SOL_SOCKET,  SO_SNDBUF,    (char *)&val, sizeof(val));
 3158    setsockopt(sock, SOL_SOCKET,  SO_RCVBUF,    (char *)&val, sizeof(val));
 3159    return sock;
 3160 }
 3161 
 3162 /*--------------------------------------------------------------------------*/
 3163 /* Write len bytes from buffer to socket sock */
 3164 static int NET_SendRaw(int sock, const void *buffer, int length, int opt)
 3165 {
 3166 
 3167   char * buf = (char *) buffer;
 3168  
 3169    int flag;
 3170    int n, nsent = 0;
 3171  
 3172    switch (opt) {
 3173    case NET_DEFAULT:
 3174      flag = 0;
 3175      break;
 3176    case NET_OOB:
 3177      flag = MSG_OOB;
 3178      break;
 3179    case NET_PEEK:            
 3180    default:
 3181      flag = 0;
 3182      break;
 3183    }
 3184  
 3185    if (sock < 0) return -1;
 3186    
 3187    for (n = 0; n < length; n += nsent) {
 3188      if ((nsent = send(sock, buf+n, length-n, flag)) <= 0) {
 3189        return nsent;
 3190      }
 3191    }
 3192 
 3193    return n;
 3194 }
 3195 
 3196 /*--------------------------------------------------------------------------*/
 3197 
 3198 static int NET_RecvRaw(int sock, void *buffer, int length)
 3199 {
 3200   /* Receive exactly length bytes into buffer. Returns number of bytes */
 3201   /* received. Returns -1 in case of error. */
 3202 
 3203 
 3204    int nrecv, n;
 3205    char *buf = (char *)buffer;
 3206 
 3207    if (sock < 0) return -1;
 3208    for (n = 0; n < length; n += nrecv) {
 3209       while ((nrecv = recv(sock, buf+n, length-n, 0)) == -1 && errno == EINTR)
 3210     errno = 0;     /* probably a SIGCLD that was caught */
 3211       if (nrecv < 0)
 3212          return nrecv;
 3213       else if (nrecv == 0)
 3214     break;        /*/ EOF */
 3215    }
 3216 
 3217    return n;
 3218 }
 3219  
 3220 /*--------------------------------------------------------------------------*/
 3221 /* Yet Another URL Parser 
 3222    url - input url
 3223    proto - input protocol
 3224    host - output host
 3225    port - output port
 3226    fn - output filename
 3227 */
 3228 
 3229 static int NET_ParseUrl(const char *url, char *proto, char *host, int *port, 
 3230          char *fn)
 3231 {
 3232   /* parses urls into their bits */
 3233   /* returns 1 if error, else 0 */
 3234 
 3235   char *urlcopy, *urlcopyorig;
 3236   char *ptrstr;
 3237   char *thost;
 3238   int isftp = 0;
 3239 
 3240   /* figure out if there is a http: or  ftp: */
 3241 
 3242   urlcopyorig = urlcopy = (char *) malloc(strlen(url)+1);
 3243   strcpy(urlcopy,url);
 3244 
 3245   /* set some defaults */
 3246   *port = 80;
 3247   strcpy(proto,"http:");
 3248   strcpy(host,"localhost");
 3249   strcpy(fn,"/");
 3250   
 3251   ptrstr = strstr(urlcopy,"http:");
 3252   if (ptrstr == NULL) {
 3253     /* Nope, not http: */
 3254     ptrstr = strstr(urlcopy,"root:");
 3255     if (ptrstr == NULL) {
 3256       /* Nope, not root either */
 3257       ptrstr = strstr(urlcopy,"ftp:");
 3258       if (ptrstr != NULL) {
 3259     if (ptrstr == urlcopy) {
 3260       strcpy(proto,"ftp:");
 3261       *port = 21;
 3262       isftp++;
 3263       urlcopy += 4; /* move past ftp: */
 3264     } else {
 3265       /* not at the beginning, bad url */
 3266       free(urlcopyorig);
 3267       return 1;
 3268     }
 3269       }
 3270     } else {
 3271       if (ptrstr == urlcopy) {
 3272     urlcopy += 5; /* move past root: */
 3273       } else {
 3274     /* not at the beginning, bad url */
 3275     free(urlcopyorig);
 3276     return 1;
 3277       }
 3278     }
 3279   } else {
 3280     if (ptrstr == urlcopy) {
 3281       urlcopy += 5; /* move past http: */
 3282     } else {
 3283       free(urlcopyorig);
 3284       return 1;
 3285     }
 3286   }
 3287 
 3288   /* got the protocol */
 3289   /* get the hostname */
 3290   if (urlcopy[0] == '/' && urlcopy[1] == '/') {
 3291     /* we have a hostname */
 3292     urlcopy += 2; /* move past the // */
 3293   }
 3294   /* do this only if http */
 3295   if (!strcmp(proto,"http:")) {
 3296 
 3297     /* Move past any user:password */
 3298     if ((thost = strchr(urlcopy, '@')) != NULL)
 3299       urlcopy = thost+1;
 3300 
 3301     if (strlen(urlcopy) > SHORTLEN-1)
 3302     {
 3303        free(urlcopyorig);
 3304        return 1;
 3305     }
 3306     strcpy(host,urlcopy);
 3307     thost = host;
 3308     while (*urlcopy != '/' && *urlcopy != ':' && *urlcopy) {
 3309       thost++;
 3310       urlcopy++;
 3311     }
 3312     /* we should either be at the end of the string, have a /, or have a : */
 3313     *thost = '\0';
 3314     if (*urlcopy == ':') {
 3315       /* follows a port number */
 3316       urlcopy++;
 3317       sscanf(urlcopy,"%d",port);
 3318       while (*urlcopy != '/' && *urlcopy) urlcopy++; /* step to the */
 3319     }
 3320   } else {
 3321     /* do this for ftp */
 3322     if (strlen(urlcopy) > SHORTLEN-1)
 3323     {
 3324        free(urlcopyorig);
 3325        return 1;
 3326     }
 3327     strcpy(host,urlcopy);
 3328     thost = host;
 3329     while (*urlcopy != '/' && *urlcopy) {
 3330       thost++;
 3331       urlcopy++; 
 3332     }
 3333     *thost = '\0';
 3334     /* Now, we should either be at the end of the string, or have a / */
 3335     
 3336   }
 3337   /* Now the rest is a fn */
 3338 
 3339   if (*urlcopy) {
 3340     if (strlen(urlcopy) > MAXLEN-1)
 3341     {
 3342        free(urlcopyorig);
 3343        return 1;
 3344     }
 3345     strcpy(fn,urlcopy);
 3346   }
 3347   free(urlcopyorig);
 3348   return 0;
 3349 }
 3350 
 3351 /*--------------------------------------------------------------------------*/
 3352 int http_checkfile (char *urltype, char *infile, char *outfile1)
 3353 {
 3354 
 3355 /* Small helper functions to set the netoutfile static string */
 3356 /* Called by cfileio after parsing the output file off of the input file url */
 3357 
 3358   char newinfile[MAXLEN];
 3359   FILE *httpfile=0;
 3360   char contentencoding[MAXLEN];
 3361   int contentlength;
 3362   int foundfile = 0;
 3363   int status=0;
 3364 
 3365   /* set defaults  */
 3366   strcpy(urltype,"http://");
 3367 
 3368   if (strlen(outfile1)) {
 3369     /* don't copy the "file://" prefix, if present.  */
 3370     if (!strncmp(outfile1, "file://", 7) ) {
 3371       strcpy(netoutfile,outfile1+7);
 3372     } else {
 3373       strcpy(netoutfile,outfile1);
 3374     }
 3375   }
 3376 
 3377   if (strstr(infile, "?")) {
 3378       /* Special case where infile name contains a "?". */
 3379       /* This is probably a CGI string; no point in testing if it exists */
 3380       /*  so just set urltype and netoutfile if necessary, then return */
 3381       
 3382       if (strlen(outfile1)) {   /* was an outfile specified? */
 3383           strcpy(urltype,"httpfile://");  
 3384 
 3385           /* don't copy the "file://" prefix, if present.  */
 3386           if (!strncmp(outfile1, "file://", 7) ) {
 3387              strcpy(netoutfile,outfile1+7);
 3388           } else {
 3389              strcpy(netoutfile,outfile1);
 3390           }
 3391       }
 3392       return 0;  /* case where infile name contains "?" */
 3393   }
 3394 
 3395   /*
 3396      If the specified infile file name does not contain a .gz or .Z suffix,
 3397      then first test if a .gz compressed version of the file exists, and if not
 3398      then test if a .Z version of the file exists. (because it will be much
 3399      faster to read the compressed file).  If the compressed files do not exist,
 3400      then finally just open the infile name exactly as specified.
 3401   */
 3402 
 3403   if (!strstr(infile,".gz") && (!strstr(infile,".Z"))) {
 3404     /* The infile string does not contain the name of a compressed file.  */
 3405     /* Fisrt, look for a .gz compressed version of the file. */
 3406     
 3407     if (strlen(infile) + 3 > MAXLEN-1)
 3408     {
 3409        return URL_PARSE_ERROR;
 3410     }  
 3411     strcpy(newinfile,infile);
 3412     strcat(newinfile,".gz");
 3413 
 3414     status = http_open_network(newinfile,&httpfile,contentencoding,
 3415                &contentlength);
 3416     if (!status) {
 3417       if (!strcmp(contentencoding, "ftp://")) {
 3418           /* this is a signal from http_open_network that indicates that */
 3419           /* the http server returned a 301 or 302 redirect to a FTP URL. */
 3420           /* Check that the file exists, because redirect many not be reliable */
 3421        
 3422           if (ftp_file_exist(newinfile)>0) { 
 3423               /* The ftp .gz compressed file is there, all is good!  */
 3424               strcpy(urltype, "ftp://");
 3425               if (strlen(newinfile) > FLEN_FILENAME-1)
 3426               {
 3427                  return URL_PARSE_ERROR;
 3428               }
 3429               strcpy(infile,newinfile);
 3430 
 3431               if (strlen(outfile1)) {
 3432                 /* there is an output file;  might need to modify the urltype */
 3433 
 3434                 if (!strncmp(outfile1, "mem:", 4) )  {
 3435                      /* copy the file to memory, with READ and WRITE access 
 3436                      In this case, it makes no difference whether the ftp file
 3437                      and or the output file are compressed or not.   */
 3438 
 3439                      strcpy(urltype, "ftpmem://");  /* use special driver */
 3440                 } else {
 3441                 /* input file is compressed */
 3442             if (strstr(outfile1,".gz") || (strstr(outfile1,".Z"))) {
 3443               strcpy(urltype,"ftpcompress://");
 3444             } else {
 3445               strcpy(urltype,"ftpfile://");
 3446             }
 3447                 } 
 3448               }
 3449 
 3450               return 0;   /* found the .gz compressed ftp file */
 3451         }
 3452             /* fall through to here if ftp redirect does not exist */
 3453       } else if (!strcmp(contentencoding, "https://")) {
 3454           /* the http server returned a 301 or 302 redirect to an HTTPS URL. */
 3455           https_checkfile(urltype, infile, outfile1);
 3456           /* For https we're not testing for compressed extensions at 
 3457              this stage.  It will all be done in https_open_network.  Therefore
 3458              leave infile alone and do immediate return. */
 3459           return 0;
 3460       } else {
 3461           /* found the http .gz compressed file */
 3462           if (httpfile)
 3463              fclose(httpfile);
 3464           foundfile = 1;
 3465           if (strlen(newinfile) > FLEN_FILENAME-1)
 3466           {
 3467              return URL_PARSE_ERROR;
 3468           }
 3469           strcpy(infile,newinfile);
 3470       }
 3471     }
 3472     else if (status != FILE_NOT_OPENED)
 3473     {
 3474        /* Some other error occured aside from not finding file, such as
 3475           a url parsing error.  Don't continue trying with other extensions. */
 3476        return status;   
 3477     }
 3478 
 3479    if (!foundfile) {
 3480     /* did not find .gz compressed version of the file, so look for .Z file. */
 3481     
 3482     if (strlen(infile+2) > MAXLEN-1)
 3483     {
 3484        return URL_PARSE_ERROR;
 3485     }  
 3486     strcpy(newinfile,infile);
 3487     strcat(newinfile,".Z");
 3488     if (!http_open_network(newinfile,&httpfile,contentencoding,
 3489                &contentlength)) {
 3490 
 3491       if (!strcmp(contentencoding, "ftp://")) {
 3492           /* this is a signal from http_open_network that indicates that */
 3493           /* the http server returned a 301 or 302 redirect to a FTP URL. */
 3494           /* Check that the file exists, because redirect many not be reliable */
 3495        
 3496           if (ftp_file_exist(newinfile)>0) { 
 3497               /* The ftp .Z compressed file is there, all is good!  */
 3498               strcpy(urltype, "ftp://");
 3499               if (strlen(newinfile) > FLEN_FILENAME-1)
 3500               {
 3501                  return URL_PARSE_ERROR;
 3502               }
 3503               strcpy(infile,newinfile);
 3504 
 3505               if (strlen(outfile1)) {
 3506                 /* there is an output file;  might need to modify the urltype */
 3507 
 3508                 if (!strncmp(outfile1, "mem:", 4) )  {
 3509                      /* copy the file to memory, with READ and WRITE access 
 3510                      In this case, it makes no difference whether the ftp file
 3511                      and or the output file are compressed or not.   */
 3512 
 3513                      strcpy(urltype, "ftpmem://");  /* use special driver */
 3514                 } else {
 3515                 /* input file is compressed */
 3516             if (strstr(outfile1,".gz") || (strstr(outfile1,".Z"))) {
 3517               strcpy(urltype,"ftpcompress://");
 3518             } else {
 3519               strcpy(urltype,"ftpfile://");
 3520             }
 3521                 } 
 3522             }
 3523             return 0;   /* found the .Z compressed ftp file */
 3524           }
 3525           /* fall through to here if ftp redirect does not exist */
 3526         }  else {
 3527            /* found the http .Z compressed file */
 3528            if (httpfile)
 3529               fclose(httpfile);
 3530            foundfile = 1;
 3531            if (strlen(newinfile) > FLEN_FILENAME-1)
 3532            {
 3533               return URL_PARSE_ERROR;
 3534            }
 3535            strcpy(infile,newinfile);
 3536         }
 3537       }
 3538     }
 3539   }  /* end of case where infile does not contain .gz or .Z */
 3540 
 3541   if (!foundfile) {
 3542     /* look for the base file.name */
 3543       
 3544     strcpy(newinfile,infile);
 3545     if (!http_open_network(newinfile,&httpfile,contentencoding,
 3546                &contentlength)) {
 3547 
 3548       if (!strcmp(contentencoding, "ftp://")) {
 3549           /* this is a signal from http_open_network that indicates that */
 3550           /* the http server returned a 301 or 302 redirect to a FTP URL. */
 3551           /* Check that the file exists, because redirect many not be reliable */
 3552        
 3553           if (ftp_file_exist(newinfile)>0) { 
 3554               /* The ftp file is there, all is good!  */
 3555               strcpy(urltype, "ftp://");
 3556               if (strlen(newinfile) > FLEN_FILENAME-1)
 3557               {
 3558                  return URL_PARSE_ERROR;
 3559               }
 3560               strcpy(infile,newinfile);
 3561 
 3562               if (strlen(outfile1)) {
 3563                 /* there is an output file;  might need to modify the urltype */
 3564 
 3565                 if (!strncmp(outfile1, "mem:", 4) )  {
 3566                      /* copy the file to memory, with READ and WRITE access 
 3567                      In this case, it makes no difference whether the ftp file
 3568                      and or the output file are compressed or not.   */
 3569 
 3570                      strcpy(urltype, "ftpmem://");  /* use special driver */
 3571                      return 0;
 3572                 } else {
 3573 
 3574               /* input file is not compressed */
 3575            strcpy(urltype,"ftpfile://");
 3576                 } 
 3577               } 
 3578               return 0;   /* found the ftp file */
 3579             }
 3580             /* fall through to here if ftp redirect does not exist */
 3581       } else if (!strcmp(contentencoding, "https://")) {
 3582           /* the http server returned a 301 or 302 redirect to an HTTPS URL. */
 3583           https_checkfile(urltype, infile, outfile1);
 3584           /* For https we're not testing for compressed extensions at 
 3585              this stage.  It will all be done in https_open_network.  Therefore
 3586              leave infile alone and do immediate return. */
 3587           return 0;
 3588       }  else {
 3589           /* found the base named file */
 3590           if (httpfile)
 3591              fclose(httpfile);
 3592           foundfile = 1;
 3593           if (strlen(newinfile) > FLEN_FILENAME-1)
 3594           {
 3595              return URL_PARSE_ERROR;
 3596           }
 3597           strcpy(infile,newinfile);
 3598       }
 3599 
 3600     }
 3601   }
 3602 
 3603   if (!foundfile) {
 3604      return (FILE_NOT_OPENED);
 3605   }
 3606 
 3607   if (strlen(outfile1)) {
 3608     /* there is an output file */
 3609 
 3610     if (!strncmp(outfile1, "mem:", 4) )  {
 3611        /* copy the file to memory, with READ and WRITE access 
 3612           In this case, it makes no difference whether the http file
 3613           and or the output file are compressed or not.   */
 3614 
 3615        strcpy(urltype, "httpmem://");  /* use special driver */
 3616        return 0;
 3617     }
 3618 
 3619     if (strstr(infile, "?")) {
 3620       /* file name contains a '?' so probably a cgi string;  */
 3621       strcpy(urltype,"httpfile://");
 3622       return 0;
 3623     }
 3624 
 3625     if (strstr(infile,".gz") || (strstr(infile,".Z"))) {
 3626     /* It's compressed */
 3627     if (strstr(outfile1,".gz") || (strstr(outfile1,".Z"))) {
 3628       strcpy(urltype,"httpcompress://");
 3629     } else {
 3630       strcpy(urltype,"httpfile://");
 3631     }
 3632     } else {
 3633     strcpy(urltype,"httpfile://");
 3634     }
 3635   } 
 3636   return 0;
 3637 }
 3638 
 3639 /*--------------------------------------------------------------------------*/
 3640 int https_checkfile (char *urltype, char *infile, char *outfile1)
 3641 {
 3642   /* set default  */
 3643   strcpy(urltype,"https://");
 3644   
 3645   if (strlen(outfile1))
 3646   {
 3647     /* don't copy the "file://" prefix, if present.  */
 3648     if (!strncmp(outfile1, "file://", 7) ) {
 3649       strcpy(netoutfile,outfile1+7);
 3650     } else {
 3651       strcpy(netoutfile,outfile1);
 3652     }
 3653     
 3654     if (!strncmp(outfile1, "mem:", 4))
 3655        strcpy(urltype,"httpsmem://");
 3656     else       
 3657        strcpy(urltype,"httpsfile://");
 3658   }
 3659 
 3660    return 0;
 3661 }
 3662 
 3663 /*--------------------------------------------------------------------------*/
 3664 int ftps_checkfile (char *urltype, char *infile, char *outfile1)
 3665 {
 3666    strcpy(urltype,"ftps://");
 3667    if (strlen(outfile1))
 3668    {
 3669      /* don't copy the "file://" prefix, if present.  */
 3670      if (!strncmp(outfile1, "file://", 7) ) {
 3671        strcpy(netoutfile,outfile1+7);
 3672      } else {
 3673        strcpy(netoutfile,outfile1);
 3674      }
 3675 
 3676      if (!strncmp(outfile1, "mem:", 4))
 3677         strcpy(urltype,"ftpsmem://");
 3678      else
 3679      {
 3680         if (strstr(outfile1,".gz") || strstr(outfile1,".Z"))
 3681         {
 3682            /* Note that for Curl dependent handlers, we can't check
 3683            at this point if infile will have a .gz or .Z appended. 
 3684            If it does not, the ftpscompress 'open' handler will fail.*/
 3685            strcpy(urltype,"ftpscompress://");
 3686         }
 3687         else
 3688            strcpy(urltype,"ftpsfile://");
 3689      }
 3690    }
 3691    return 0;
 3692 }
 3693 
 3694 /*--------------------------------------------------------------------------*/
 3695 int ftp_checkfile (char *urltype, char *infile, char *outfile1)
 3696 {
 3697   char newinfile[MAXLEN];
 3698   FILE *ftpfile;
 3699   FILE *command;
 3700   int sock;
 3701   int foundfile = 0;
 3702   int status=0;
 3703 
 3704  /* Small helper functions to set the netoutfile static string */
 3705 
 3706   /* default to ftp://  if no outfile specified */
 3707   strcpy(urltype,"ftp://"); 
 3708 
 3709  if (!strstr(infile,".gz") && (!strstr(infile,".Z"))) {
 3710     /* The infile string does not contain the name of a compressed file.  */
 3711     /* Fisrt, look for a .gz compressed version of the file. */
 3712       
 3713     if (strlen(infile)+3 > MAXLEN-1)
 3714     {
 3715        return URL_PARSE_ERROR;
 3716     }
 3717     strcpy(newinfile,infile);
 3718     strcat(newinfile,".gz");
 3719  
 3720     /* look for .gz version of the file */
 3721     status = ftp_file_exist(newinfile);
 3722     if (status > 0) {
 3723       foundfile = 1;
 3724       if (strlen(newinfile) > FLEN_FILENAME-1)
 3725          return URL_PARSE_ERROR;
 3726       strcpy(infile,newinfile);
 3727     }
 3728     else if (status < 0)
 3729     {
 3730        /* Server is demanding an SSL connection. 
 3731           Change urltype and exit. */
 3732        ftps_checkfile(urltype, infile, outfile1);
 3733        return 0;
 3734     }
 3735 
 3736     if (!foundfile) {
 3737       if (strlen(infile)+2 > MAXLEN-1)
 3738       {
 3739          return URL_PARSE_ERROR;
 3740       }
 3741       strcpy(newinfile,infile);
 3742       strcat(newinfile,".Z");
 3743  
 3744     /* look for .Z version of the file */
 3745       if (ftp_file_exist(newinfile)) {
 3746         foundfile = 1;
 3747         if (strlen(newinfile) > FLEN_FILENAME-1)
 3748            return URL_PARSE_ERROR;
 3749         strcpy(infile,newinfile);
 3750       }
 3751     }
 3752   }
 3753 
 3754   if (!foundfile) {
 3755       strcpy(newinfile,infile);
 3756  
 3757       /* look for the base file */
 3758       status = ftp_file_exist(newinfile);
 3759       if (status > 0) {
 3760         foundfile = 1;
 3761         if (strlen(newinfile) > FLEN_FILENAME-1)
 3762            return URL_PARSE_ERROR;
 3763         strcpy(infile,newinfile);
 3764       }
 3765       else if (status < 0)
 3766       {
 3767          /* Server is demanding an SSL connection. 
 3768             Change urltype and exit. */
 3769          ftps_checkfile(urltype, infile, outfile1);
 3770          return 0;
 3771       }
 3772   }
 3773 
 3774   if (!foundfile) {
 3775      return (FILE_NOT_OPENED);
 3776   }
 3777 
 3778   if (strlen(outfile1)) {
 3779     /* there is an output file;  might need to modify the urltype */
 3780 
 3781     /* don't copy the "file://" prefix, if present.  */
 3782     if (!strncmp(outfile1, "file://", 7) )
 3783        strcpy(netoutfile,outfile1+7);
 3784     else
 3785        strcpy(netoutfile,outfile1);
 3786 
 3787     if (!strncmp(outfile1, "mem:", 4) )  {
 3788        /* copy the file to memory, with READ and WRITE access 
 3789           In this case, it makes no difference whether the ftp file
 3790           and or the output file are compressed or not.   */
 3791 
 3792        strcpy(urltype, "ftpmem://");  /* use special driver */
 3793        return 0;
 3794     }
 3795  
 3796     if (strstr(infile,".gz") || (strstr(infile,".Z"))) {
 3797     /* input file is compressed */
 3798     if (strstr(outfile1,".gz") || (strstr(outfile1,".Z"))) {
 3799       strcpy(urltype,"ftpcompress://");
 3800     } else {
 3801       strcpy(urltype,"ftpfile://");
 3802     }
 3803     } else {
 3804     strcpy(urltype,"ftpfile://");
 3805     } 
 3806   } 
 3807   return 0;
 3808 }
 3809 /*--------------------------------------------------------------------------*/
 3810 /* A small helper function to wait for a particular status on the ftp 
 3811    connectino */
 3812 static int ftp_status(FILE *ftp, char *statusstr)
 3813 {
 3814   /* read through until we find a string beginning with statusstr */
 3815   /* This needs a timeout */
 3816   
 3817   /* Modified 2/19 to return the numerical value of the returned status when
 3818      it differs from the requested status. */
 3819 
 3820   char recbuf[MAXLEN], errorstr[SHORTLEN];
 3821   int len, ftpcode=0;
 3822 
 3823   len = strlen(statusstr);
 3824   while (1) {
 3825 
 3826     if (!(fgets(recbuf,MAXLEN,ftp))) {
 3827       snprintf(errorstr,SHORTLEN,"ERROR: ftp_status wants %s but fgets returned 0",statusstr);
 3828       ffpmsg(errorstr);
 3829       return 1; /* error reading */
 3830     }
 3831 
 3832     recbuf[len] = '\0'; /* make it short */
 3833     if (!strcmp(recbuf,statusstr)) {
 3834       return 0; /* we're ok */
 3835     }
 3836     if (recbuf[0] > '3') {
 3837       /* oh well, some sort of error. */
 3838       snprintf(errorstr,SHORTLEN,"ERROR ftp_status wants %s but got %s", statusstr, recbuf);
 3839       ffpmsg(errorstr);
 3840       /* Return the numerical code, if string can be converted to int.
 3841          But must not return 0 from here. */
 3842       ftpcode = atoi(recbuf);
 3843       return ftpcode ? ftpcode : 1; 
 3844     }
 3845     snprintf(errorstr,SHORTLEN,"ERROR ftp_status wants %s but got unexpected %s", statusstr, recbuf);
 3846     ffpmsg(errorstr);
 3847   }
 3848 }
 3849 
 3850 /*
 3851  *----------------------------------------------------------------------
 3852  *
 3853  * CreateSocketAddress --
 3854  *
 3855  *  This function initializes a sockaddr structure for a host and port.
 3856  *
 3857  * Results:
 3858  *  1 if the host was valid, 0 if the host could not be converted to
 3859  *  an IP address.
 3860  *
 3861  * Side effects:
 3862  *  Fills in the *sockaddrPtr structure.
 3863  *
 3864  *----------------------------------------------------------------------
 3865  */
 3866 
 3867 static int
 3868 CreateSocketAddress(
 3869     struct sockaddr_in *sockaddrPtr,    /* Socket address */
 3870     char *host,             /* Host.  NULL implies INADDR_ANY */
 3871     int port)               /* Port number */
 3872 {
 3873     struct hostent *hostent;        /* Host database entry */
 3874     struct in_addr addr;        /* For 64/32 bit madness */
 3875     char localhost[MAXLEN];
 3876 
 3877     strcpy(localhost,host);
 3878 
 3879     memset((void *) sockaddrPtr, '\0', sizeof(struct sockaddr_in));
 3880     sockaddrPtr->sin_family = AF_INET;
 3881     sockaddrPtr->sin_port = htons((unsigned short) (port & 0xFFFF));
 3882     if (host == NULL) {
 3883     addr.s_addr = INADDR_ANY;
 3884     } else {
 3885         addr.s_addr = inet_addr(localhost);
 3886         if (addr.s_addr == 0xFFFFFFFF) {
 3887             hostent = gethostbyname(localhost);
 3888             if (hostent != NULL) {
 3889                 memcpy((void *) &addr,
 3890                         (void *) hostent->h_addr_list[0],
 3891                         (size_t) hostent->h_length);
 3892             } else {
 3893 #ifdef  EHOSTUNREACH
 3894                 errno = EHOSTUNREACH;
 3895 #else
 3896 #ifdef ENXIO
 3897                 errno = ENXIO;
 3898 #endif
 3899 #endif
 3900                 return 0;   /* error */
 3901             }
 3902         }
 3903     }
 3904         
 3905     /*
 3906      * NOTE: On 64 bit machines the assignment below is rumored to not
 3907      * do the right thing. Please report errors related to this if you
 3908      * observe incorrect behavior on 64 bit machines such as DEC Alphas.
 3909      * Should we modify this code to do an explicit memcpy?
 3910      */
 3911 
 3912     sockaddrPtr->sin_addr.s_addr = addr.s_addr;
 3913     return 1;   /* Success. */
 3914 }
 3915 
 3916 /* Signal handler for timeouts */
 3917 
 3918 static void signal_handler(int sig) {
 3919 
 3920   switch (sig) {
 3921   case SIGALRM:    /* process for alarm */
 3922     longjmp(env,sig);
 3923     
 3924   default: {
 3925       /* Hmm, shouldn't have happend */
 3926       exit(sig);
 3927     }
 3928   }
 3929 }
 3930 
 3931 /**************************************************************/
 3932 
 3933 /* Root driver */
 3934 
 3935 /*--------------------------------------------------------------------------*/
 3936 int root_init(void)
 3937 {
 3938     int ii;
 3939 
 3940     for (ii = 0; ii < NMAXFILES; ii++) /* initialize all empty slots in table */
 3941     {
 3942        handleTable[ii].sock = 0;
 3943        handleTable[ii].currentpos = 0;
 3944     }
 3945     return(0);
 3946 }
 3947 /*--------------------------------------------------------------------------*/
 3948 int root_setoptions(int options)
 3949 {
 3950   /* do something with the options argument, to stop compiler warning */
 3951   options = 0;
 3952   return(options);
 3953 }
 3954 /*--------------------------------------------------------------------------*/
 3955 int root_getoptions(int *options)
 3956 {
 3957   *options = 0;
 3958   return(0);
 3959 }
 3960 /*--------------------------------------------------------------------------*/
 3961 int root_getversion(int *version)
 3962 {
 3963     *version = 10;
 3964     return(0);
 3965 }
 3966 /*--------------------------------------------------------------------------*/
 3967 int root_shutdown(void)
 3968 {
 3969   return(0);
 3970 }
 3971 /*--------------------------------------------------------------------------*/
 3972 int root_open(char *url, int rwmode, int *handle)
 3973 {
 3974     int ii, status;
 3975     int sock;
 3976 
 3977     *handle = -1;
 3978     for (ii = 0; ii < NMAXFILES; ii++)  /* find empty slot in table */
 3979     {
 3980         if (handleTable[ii].sock == 0)
 3981         {
 3982             *handle = ii;
 3983             break;
 3984         }
 3985     }
 3986 
 3987     if (*handle == -1)
 3988        return(TOO_MANY_FILES);    /* too many files opened */
 3989 
 3990     /*open the file */
 3991     if (rwmode) {
 3992       status = root_openfile(url, "update", &sock);
 3993     } else {
 3994       status = root_openfile(url, "read", &sock);
 3995     }
 3996     if (status)
 3997       return(status);
 3998     
 3999     handleTable[ii].sock = sock;
 4000     handleTable[ii].currentpos = 0;
 4001     
 4002     return(0);
 4003 }
 4004 /*--------------------------------------------------------------------------*/
 4005 int root_create(char *filename, int *handle)
 4006 {
 4007     int ii, status;
 4008     int sock;
 4009 
 4010     *handle = -1;
 4011     for (ii = 0; ii < NMAXFILES; ii++)  /* find empty slot in table */
 4012     {
 4013         if (handleTable[ii].sock == 0)
 4014         {
 4015             *handle = ii;
 4016             break;
 4017         }
 4018     }
 4019 
 4020     if (*handle == -1)
 4021        return(TOO_MANY_FILES);    /* too many files opened */
 4022 
 4023     /*open the file */
 4024     status = root_openfile(filename, "create", &sock);
 4025 
 4026     if (status) {
 4027       ffpmsg("Unable to create file");
 4028       return(status);
 4029     }
 4030     
 4031     handleTable[ii].sock = sock;
 4032     handleTable[ii].currentpos = 0;
 4033     
 4034     return(0);
 4035 }
 4036 /*--------------------------------------------------------------------------*/
 4037 int root_size(int handle, LONGLONG *filesize)
 4038 /*
 4039   return the size of the file in bytes
 4040 */
 4041 {
 4042 
 4043   int sock;
 4044   int offset;
 4045   int status;
 4046   int op;
 4047 
 4048   sock = handleTable[handle].sock;
 4049 
 4050   status = root_send_buffer(sock,ROOTD_STAT,NULL,0);
 4051   status = root_recv_buffer(sock,&op,(char *)&offset, 4);
 4052   *filesize = (LONGLONG) ntohl(offset);
 4053   
 4054   return(0);
 4055 }
 4056 /*--------------------------------------------------------------------------*/
 4057 int root_close(int handle)
 4058 /*
 4059   close the file
 4060 */
 4061 {
 4062 
 4063   int status;
 4064   int sock;
 4065 
 4066   sock = handleTable[handle].sock;
 4067   status = root_send_buffer(sock,ROOTD_CLOSE,NULL,0);
 4068   close(sock);
 4069   handleTable[handle].sock = 0;
 4070   return(0);
 4071 }
 4072 /*--------------------------------------------------------------------------*/
 4073 int root_flush(int handle)
 4074 /*
 4075   flush the file
 4076 */
 4077 {
 4078   int status;
 4079   int sock;
 4080 
 4081   sock = handleTable[handle].sock;
 4082   status = root_send_buffer(sock,ROOTD_FLUSH,NULL,0);
 4083   return(0);
 4084 }
 4085 /*--------------------------------------------------------------------------*/
 4086 int root_seek(int handle, LONGLONG offset)
 4087 /*
 4088   seek to position relative to start of the file
 4089 */
 4090 {
 4091   handleTable[handle].currentpos = offset;
 4092   return(0);
 4093 }
 4094 /*--------------------------------------------------------------------------*/
 4095 int root_read(int hdl, void *buffer, long nbytes)
 4096 /*
 4097   read bytes from the current position in the file
 4098 */
 4099 {
 4100   char msg[SHORTLEN];
 4101   int op;
 4102   int status;
 4103   int astat;
 4104 
 4105   /* we presume here that the file position will never be > 2**31 = 2.1GB */
 4106   snprintf(msg,SHORTLEN,"%ld %ld ",(long) handleTable[hdl].currentpos,nbytes);
 4107   status = root_send_buffer(handleTable[hdl].sock,ROOTD_GET,msg,strlen(msg));
 4108   if ((unsigned) status != strlen(msg)) {
 4109     return (READ_ERROR);
 4110   }
 4111   astat = 0;
 4112   status = root_recv_buffer(handleTable[hdl].sock,&op,(char *) &astat,4);
 4113   if (astat != 0) {
 4114     return (READ_ERROR);
 4115   }
 4116 
 4117   status = NET_RecvRaw(handleTable[hdl].sock,buffer,nbytes);
 4118   if (status != nbytes) {
 4119     return (READ_ERROR);
 4120   }
 4121   handleTable[hdl].currentpos += nbytes;
 4122 
 4123   return(0);
 4124 }
 4125 /*--------------------------------------------------------------------------*/
 4126 int root_write(int hdl, void *buffer, long nbytes)
 4127 /*
 4128   write bytes at the current position in the file
 4129 */
 4130 {
 4131 
 4132   char msg[SHORTLEN];
 4133   int len;
 4134   int sock;
 4135   int status;
 4136   int astat;
 4137   int op;
 4138 
 4139   sock = handleTable[hdl].sock;
 4140   /* we presume here that the file position will never be > 2**31 = 2.1GB */
 4141   snprintf(msg,SHORTLEN,"%ld %ld ",(long) handleTable[hdl].currentpos,nbytes);
 4142 
 4143   len = strlen(msg);
 4144   status = root_send_buffer(sock,ROOTD_PUT,msg,len+1);
 4145   if (status != len+1) {
 4146     return (WRITE_ERROR);
 4147   }
 4148   status = NET_SendRaw(sock,buffer,nbytes,NET_DEFAULT);
 4149   if (status != nbytes) {
 4150     return (WRITE_ERROR);
 4151   }
 4152   astat = 0;
 4153   status = root_recv_buffer(handleTable[hdl].sock,&op,(char *) &astat,4);
 4154 
 4155   if (astat != 0) {
 4156     return (WRITE_ERROR);
 4157   }
 4158   handleTable[hdl].currentpos += nbytes;
 4159   return(0);
 4160 }
 4161 
 4162 /*--------------------------------------------------------------------------*/
 4163 int root_openfile(char *url, char *rwmode, int *sock)
 4164      /*
 4165        lowest level routine to physically open a root file
 4166      */
 4167 {
 4168   
 4169   int status;
 4170   char recbuf[MAXLEN];
 4171   char errorstr[MAXLEN];
 4172   char proto[SHORTLEN];
 4173   char host[SHORTLEN];
 4174   char fn[MAXLEN];
 4175   char turl[MAXLEN];
 4176   int port;
 4177   int op;
 4178   int ii;
 4179   int authstat;
 4180   
 4181   
 4182   /* Parse the URL apart again */
 4183   if (strlen(url)+7 > MAXLEN-1)
 4184   {
 4185      ffpmsg("Error: url too long");
 4186      return(FILE_NOT_OPENED);
 4187   }
 4188   strcpy(turl,"root://");
 4189   strcat(turl,url);
 4190   if (NET_ParseUrl(turl,proto,host,&port,fn)) {
 4191     snprintf(errorstr,MAXLEN,"URL Parse Error (root_open) %s",url);
 4192     ffpmsg(errorstr);
 4193     return (FILE_NOT_OPENED);
 4194   }
 4195   
 4196   /* Connect to the remote host */
 4197   *sock = NET_TcpConnect(host,port);
 4198   if (*sock < 0) {
 4199     ffpmsg("Couldn't connect to host (root_openfile)");
 4200     return (FILE_NOT_OPENED);
 4201   }
 4202   
 4203   /* get the username */
 4204   if (NULL != getenv("ROOTUSERNAME")) {
 4205     if (strlen(getenv("ROOTUSERNAME")) > MAXLEN-1)
 4206     {
 4207        ffpmsg("root user name too long (root_openfile)");
 4208        return (FILE_NOT_OPENED);
 4209     }
 4210     strcpy(recbuf,getenv("ROOTUSERNAME"));
 4211   } else {
 4212     printf("Username: ");
 4213     fgets(recbuf,MAXLEN,stdin);
 4214     recbuf[strlen(recbuf)-1] = '\0';
 4215   }
 4216   
 4217   status = root_send_buffer(*sock, ROOTD_USER, recbuf,strlen(recbuf));
 4218   if (status < 0) {
 4219     ffpmsg("error talking to remote system on username ");
 4220     return (FILE_NOT_OPENED);
 4221   }
 4222   
 4223   status = root_recv_buffer(*sock,&op,(char *)&authstat,4);
 4224   if (!status) {
 4225     ffpmsg("error talking to remote system on username");
 4226     return (FILE_NOT_OPENED);
 4227   }
 4228   
 4229   if (op != ROOTD_AUTH) {
 4230     ffpmsg("ERROR on ROOTD_USER");
 4231     ffpmsg(recbuf);
 4232     return (FILE_NOT_OPENED);
 4233   }
 4234   
 4235 
 4236   /* now the password */
 4237   if (NULL != getenv("ROOTPASSWORD")) {
 4238     if (strlen(getenv("ROOTPASSWORD")) > MAXLEN-1)
 4239     {
 4240        ffpmsg("root password too long (root_openfile)");
 4241        return (FILE_NOT_OPENED);
 4242     }
 4243     strcpy(recbuf,getenv("ROOTPASSWORD"));
 4244   } else {
 4245     printf("Password: ");
 4246     fgets(recbuf,MAXLEN,stdin);
 4247     recbuf[strlen(recbuf)-1] = '\0';
 4248   }
 4249   /* ones complement the password */
 4250   for (ii=0;(unsigned) ii<strlen(recbuf);ii++) {
 4251     recbuf[ii] = ~recbuf[ii];
 4252   }
 4253   
 4254   status = root_send_buffer(*sock, ROOTD_PASS, recbuf, strlen(recbuf));
 4255   if (status < 0) {
 4256     ffpmsg("error talking to remote system sending password");
 4257     return (FILE_NOT_OPENED);
 4258   }
 4259   
 4260   status = root_recv_buffer(*sock,&op,(char *)&authstat,4);
 4261   if (status < 0) {
 4262     ffpmsg("error talking to remote system acking password");
 4263     return (FILE_NOT_OPENED);
 4264   }
 4265   
 4266   if (op != ROOTD_AUTH) {
 4267     ffpmsg("ERROR on ROOTD_PASS");
 4268     ffpmsg(recbuf);
 4269     return (FILE_NOT_OPENED);
 4270   }
 4271   
 4272   /* now the file open request */
 4273   if (strlen(fn)+strlen(rwmode)+1 > MAXLEN-1)
 4274   {
 4275      ffpmsg("root file name too long (root_openfile)");
 4276      return (FILE_NOT_OPENED);
 4277   }
 4278   strcpy(recbuf,fn);
 4279   strcat(recbuf," ");
 4280   strcat(recbuf,rwmode);
 4281 
 4282   status = root_send_buffer(*sock, ROOTD_OPEN, recbuf, strlen(recbuf));
 4283   if (status < 0) {
 4284     ffpmsg("error talking to remote system on open ");
 4285     return (FILE_NOT_OPENED);
 4286   }
 4287 
 4288   status = root_recv_buffer(*sock,&op,(char *)&authstat,4);
 4289   if (status < 0) {
 4290     ffpmsg("error talking to remote system on open");
 4291     return (FILE_NOT_OPENED);
 4292   }
 4293   
 4294   if ((op != ROOTD_OPEN) && (authstat != 0)) {
 4295     ffpmsg("ERROR on ROOTD_OPEN");
 4296     ffpmsg(recbuf);
 4297     return (FILE_NOT_OPENED);
 4298   }
 4299 
 4300   return 0;
 4301 
 4302 }
 4303 
 4304 static int root_send_buffer(int sock, int op, char *buffer, int buflen)
 4305 {
 4306   /* send a buffer, the form is
 4307      <len>
 4308      <op>
 4309      <buffer>
 4310 
 4311      <len> includes the 4 bytes for the op, the length bytes (4) are implicit
 4312 
 4313 
 4314      if buffer is null don't send it, not everything needs something sent */
 4315 
 4316   int len;
 4317   int status;
 4318 
 4319   int hdr[2];
 4320 
 4321   len = 4;
 4322 
 4323   if (buffer != NULL) {
 4324     len += buflen;
 4325   }
 4326   
 4327   hdr[0] = htonl(len);
 4328   hdr[1] = htonl(op);
 4329 
 4330   status = NET_SendRaw(sock,hdr,sizeof(hdr),NET_DEFAULT);
 4331   if (status < 0) {
 4332     return status;
 4333   }
 4334   if (buffer != NULL) {
 4335     status = NET_SendRaw(sock,buffer,buflen,NET_DEFAULT);
 4336   }
 4337   return status;
 4338 }
 4339   
 4340 static int root_recv_buffer(int sock, int *op, char *buffer, int buflen)
 4341 {
 4342   /* recv a buffer, the form is
 4343      <len>
 4344      <op>
 4345      <buffer>
 4346   */
 4347 
 4348   int recv1 = 0;
 4349   int len;
 4350   int status;
 4351   char recbuf[MAXLEN];
 4352 
 4353   status = NET_RecvRaw(sock,&len,4);
 4354 
 4355   if (status < 0) {
 4356     return status;
 4357   }
 4358   recv1 += status;
 4359 
 4360   len = ntohl(len);
 4361 
 4362   /* ok, have the length, recive the operation */
 4363   len -= 4;
 4364   status = NET_RecvRaw(sock,op,4);
 4365   if (status < 0) {
 4366     return status;
 4367   }
 4368 
 4369   recv1 += status;
 4370 
 4371   *op = ntohl(*op);
 4372   
 4373   if (len > MAXLEN) {
 4374     len = MAXLEN;
 4375   }
 4376 
 4377   if (len > 0) { /* Get the rest of the message */
 4378     status = NET_RecvRaw(sock,recbuf,len);
 4379     if (len > buflen) {
 4380       len = buflen;
 4381     }
 4382     memcpy(buffer,recbuf,len);
 4383     if (status < 0) {
 4384       return status;
 4385     }
 4386   } 
 4387 
 4388   recv1 += status;
 4389   return recv1;
 4390 
 4391 }
 4392 
 4393 /*****************************************************************************/
 4394 /*
 4395   Encode a string into MIME Base64 format string
 4396 */
 4397 
 4398 
 4399 static int encode64(unsigned s_len, char *src, unsigned d_len, char *dst) {
 4400 
 4401   static char base64[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
 4402 "abcdefghijklmnopqrstuvwxyz"
 4403 "0123456789"
 4404 "+/";
 4405 
 4406   unsigned triad;
 4407 
 4408 
 4409   for (triad = 0; triad < s_len; triad += 3) {
 4410     unsigned long int sr;
 4411     unsigned byte;
 4412 
 4413     for (byte = 0; (byte<3) && (triad+byte<s_len); ++byte) {
 4414       sr <<= 8;
 4415       sr |= (*(src+triad+byte) & 0xff);
 4416     }
 4417 
 4418     /* shift left to next 6 bit alignment*/
 4419     sr <<= (6-((8*byte)%6))%6;
 4420 
 4421     if (d_len < 4)
 4422       return 1;
 4423 
 4424     *(dst+0) = *(dst+1) = *(dst+2) = *(dst+3) = '=';
 4425     switch(byte) {
 4426     case 3:
 4427       *(dst+3) = base64[sr&0x3f];
 4428       sr >>= 6;
 4429     case 2:
 4430       *(dst+2) = base64[sr&0x3f];
 4431       sr >>= 6;
 4432     case 1:
 4433       *(dst+1) = base64[sr&0x3f];
 4434       sr >>= 6;
 4435       *(dst+0) = base64[sr&0x3f];
 4436     }
 4437     dst += 4;
 4438     d_len -= 4;
 4439   }
 4440 
 4441   *dst = '\0';
 4442   return 0;
 4443 }
 4444 
 4445 #endif