SOPALE Overview DMTM (Deep Model ThermoMechanical) version (code from November 2000) April 12, 2000 Input ===== GENERAL REMARKS ABOUT INPUT FILES Software written by Philippe uses input files which are ASCII (text) files, created and edited using your favourite text editor. Philippe's software doesn't look for special characters which delimit comments. So we put any comments at the end of a line of input, and rely on the Fortran READ statement to ignore any characters after it has read the expected values from a line of input. AUXILIARY INPUT FILES Main input files for dmtm SOPALE: SOPALE1_i - specifies most of the parameters for the model sopale1_plus_i - additional file containing parameters for grid generation and thermal layers erode_i - parameters for erosion The input parameters for these three files are explained in the file "sopale_inputs.txt." There are also extra input files which Philippe created to specify other options as SOPALE evolved. They deal with features that are of interest to Philippe. Usually we just copy the same ones from one run to the next, since we rarely change them. They are listed below. SOPALE1_interactive_graphics_i analyze_wedge_growth_i check_picard_i check_picard_show ********************** Input file descriptions *********************** 1. SOPALE1_i This is the main input file for SOPALE. It's a text file, edited with the text editor of your choice. The contents of this file is described in the file "sopale_inputs.txt". It's best to start from an existing file and modify it to suit your needs, rather than try to create one from scratch. 2. sopale1_plus_i This file contains parameters for grid generation and thermal layers. See "sopale_inputs.txt" for a description of this file. 3. erode_i This file contains the erosion parameters. It's required even if you're not doing any erosion in your model. See "sopale_inputs.txt" for a description of this file. 4. SOPALE_interactive_graphics_i Contains parameters relating to the screen display during a SOPALE run. We (Chris and I) use a version of the code which does not display to screen as it runs, so this input file is kind of irrelevant. However, the code requires this file to exist, even it it doesn't use it. Example of contents: 100 100 100 100 100 100 100 ! show_eg show_ec show_ev show_es show_esr show_et 10 10 10 !show_lg show_lc show_ls 0 0. ! vopt 0=show velocity 1=show error velocity vectors 1 1 ! ix,iy (spacing) 0 ! iwantprints 0=no postscript, 1=generate postscript 0. 40. -20. 20. ! xmin1,xmax1,zmin1,zmax1 (size of plot window) user units 0 0 800 800 ! w1,w2,w3,w4 (w1,w2) is upper left corner of window; w3=width,w4=height (pixels) 0.1 0.005 ! vdt=multiplier for vectors; radius=radius of disc at vector origin 2 ! pcolors= number of intervals used to color plot elemental fields <= 228 0 1.2 60000 0 0 !fp1(i),fp2(i),pr(i),pg(i),pb(i); left bound for interval, right bound, rgb values (0-65536) 1.3 2.3 0 60000 0 .1 50 3 ! cprecis (use between .1 and .5), pcontours=number of contour intervals (when contour_style=0), whatf 1=pressure 2=tau_xy 3=deviatoric stress(2nd inv) -1 ! contour_style 0=autoscale 1=given values (-1=no contours) ! ncontourv (only if contour_style=1) number of contour values ! contourv(i),i=1,ncontourv - contour values 5. analyze_wedge_growth_i This is one of those auxiliary files that Philippe uses. It's required by the code, but we don't usually change it (just copy it from some previous run). Example contents: 10000 998 201 101 ! first number is n - every n timesteps, produce output for postprocessor analyzewedge.f 6. check_picard_i This is one of those auxiliary files that Philippe uses. It's required by the code, but we don't usually change it (just copy it from some previous run). Example contents: -1 0 1 1.01 1.02 1.05 1.1 1.2 1.4 2. 5. 1.d30 1. 7. check_picard_show This is one of those auxiliary files that Philippe uses. It's required by the code, but we don't usually change it (just copy it from some previous run). Example contents: 0 8. Matrix files These files describe the grid used by a SOPALE run. The name of the matrix file is specified in the SOPALE1_i input file (see the file "sopale_inputs.txt" for a description of SOPALE1_i inputs). The matrix files are created by a program called pmain.x. The source code for pmain.x is in the file pmain.f and can be found in the "matrices" subdirectory. To create a new matrix file, you need to edit two statements in the source code pmain.f. (1) In pmain.f, look for the PARAMETER statement (around line 16): parameter (ndof=2,ndim=2,nx=801,ny=41,nz=1,neq=ndof*nx*ny*nz, 1nb=20,ll=nb*ndof,nne=4,nn=nx*ny*nz,nzamax=1600000) Change nx to be the number of columns in your grid, change ny to be the number of rows in your grid. Change ndof to an appropriate value for your model: 2 for mechanical systems, 1 for thermal systems. (2) Find the OPEN statement for unit 1 (around line 66): open(unit=1,file='umat801.41') Change the filename to be consistent with your nx and ny values. The convention for naming the matrix files is "umatXX.YY" where XX is the nx value and YY is the ny value. After editing these two statements, save the program, compile and run it. On the SP2, you can use this commands to compile: xlf pmain.f -o pmain.x and then run it by typing: ./pmain.x Output ====== SOPALE produces various binary files as output. The main ones we deal with are the ones with names like: g01_p00_f01_o g02_p00_f01_o g01_p00_f02_o g02_p00_f02_o These files are named gGG_pPP_fNN_o where GG is the grid number, 01 = Eulerian grid 02 = Lagrangian grid PP is the processor number, used with parallel programs to indicate which processor produced this output file. In the case of SOPALE it will always be 00 because SOPALE is a serial program and uses only one processor. NN is the "frame" number and indicates indirectly which timestep the output depicts. If you have indicated in your SOPALE1_i input file that you want output every 100 timesteps, then the first output file, generated at timestep 100, will have frame number 01. The second output file, in this case generated at timestep 200, will have frame number 02, and so on. There are other files produced as output, but these are the main ones which you would be interested in viewing. You can view the output files using a postprocessing program called "mozart_plot.x", written by Philippe Fullsack. A user's guide to the postprocessing program is in the file "sopale_postproc.txt". Directory structure/setup ========================= 1) The usual setup (designed by Russ): Each model run will have its own directory, which we will call "the model directory". Model directory contains these files: SOPALE1_i - main input text file sopale1_plus_i - grid generation input file erode_i - erosion input file SOPALE1.x - Sopale executable (may have a different name) SOPALE1_interactive_graphics_i analyze_wedge_growth_i check_picard_i check_picard_show When you run the model, the matrix file (named something like 'umat801.41') will be placed in the model directory. Model directory also contains the subdirectory "Sopale", which in turn contains the subdirectory "r30", which is where the output of the model run will go. (There is no particular reason why the output directory is named "r30", aside from historical reasons...) Tree Diagram for typical setup: Model directory (should have a name descriptive of the model) | +- SOPALE1.x +- SOPALE1_i +- SOPALE_interactive_graphics_i +- check_picard_i +- check_picard_show +- analyze_wedge_growth_i | +- matrix file (e.g. 'umat801.41') - this is copied here during | a model run | +- Sopale (subdirectory) | +- r30 (subdirectory for output) | +- g01_p00_f01_o } +- g02_p00_f01_o } These are generated +- g01_p00_f02_o } during the model run. +- g02_p00_f02_o } +- ... other output files } 2) Variation used in older setup (included here only to help you find your way around some of the older SOPALE runs): We put the input files (SOPALE1_i, SOPALE_interactive_graphics_i, analyze_wedge_growth_i, check_picard_i, check_picard_show) in same directory where SOPALE will write the output files. We had to use some strange-looking pathnames in the SOPALE1_i input file to accomplish this (see the "sopale_inputs.txt" file) but it makes it easier to keep different model runs organized in the long run. In the diagram below, the directories for each run are named "r01", "r02", and so on, but they could be named whatever you like. For this way of setting things up, you are not restricted to 3-character directory names. Tree Diagram for Chris and Bonny's old setup: main Sopale directory | +- SOPALE1np.x (Sopale executable) | +- matrices (subdirectory) | | | +- matrix files | +- pmain.f | +- pmain.x | +- Sopale (subdirectory) | +- r01 (subdirectory for output of a model run) | | | +- SOPALE1_i ) | +- SOPALE_interactive_graphics_i ) | +- check_picard_i ) Input files for this | +- check_picard_show ) model run | +- analyze_wedge_growth_i ) | +- g01_p00_f01_o } | +- g02_p00_f01_o } | +- g01_p00_f02_o } Output files for this | +- g02_p00_f02_o } model run | +- ... other output files } | +- r02 (subdirectory for output of a model run) | | | +- SOPALE1_i ) | +- SOPALE_interactive_graphics_i ) | +- check_picard_i ) Input files for this | +- check_picard_show ) model run | +- analyze_wedge_growth_i ) | +- g01_p00_f01_o } | +- g02_p00_f01_o } | +- g01_p00_f02_o } Output files for this | +- g02_p00_f02_o } model run | +- ... other output files } | +- r03 (subdirectory for output of a model run) | | | +- SOPALE1_i ) | +- SOPALE_interactive_graphics_i ) | +- check_picard_i ) Input files for this | +- check_picard_show ) model run | +- analyze_wedge_growth_i ) | +- g01_p00_f01_o } | +- g02_p00_f01_o } | +- g01_p00_f02_o } Output files for this | +- g02_p00_f02_o } model run | +- ... other output files } | . . etc. . | +- r30 (subdirectory for output of a model run) | +- SOPALE1_i ) +- SOPALE_interactive_graphics_i ) +- check_picard_i ) Input files for this +- check_picard_show ) model run +- analyze_wedge_growth_i ) +- g01_p00_f01_o } +- g02_p00_f01_o } +- g01_p00_f02_o } Output files for this +- g02_p00_f02_o } model run +- ... other output files } ***************** How to run SOPALE ***************** If you were running SOPALE at the command line, you would just edit your input file (SOPALE1_i), then type "sopale1.x" (or whatever you've named your Sopale executable) to run the program and that would be it. Sopale would generate some output, you would copy the output to wherever you want to postprocess it, and then run the postprocessing program. However, we don't usually run SOPALE at the command line, since SOPALE runs can be CPU-intensive and take a while to run. Instead, we create a batch file of commands and submit this to the Loadleveler. The Loadleveler is a program which distributes jobs among the various nodes of the SP2. Here is an example of a simple batch file for the Loadleveler: #!/bin/sh #@ output = out #@ error = err #@ notification = Always #@ job_type = serial #@ class = long #@ queue ./SOPALE1np.x The lines starting with "#@" are options or commands for the Loadleveler. The other lines are shell command lines. #!/bin/sh This is a shell script command which means "use the shell /bin/sh to execute these commands" #@ output = out This is a Loadleveler option which sends the standard output to a file called 'out'. You can change the name of this file to something else if you like. #@ error = err A Loadleveler option which sends the error messages to a file called 'err'. You can change the name of this file to something else if you like. #@ notification = Always A Loadleveler option which sends you email when your job starts and when it finishes. The email will be sent to your SP2 account, but you can set your SP2 email to be forwarded to another account. #@ job_type = serial This tells the Loadleveler that the job is serial, not parallel. #@ class = long This tells the Loadleveler that you don't want to impose any CPU time limit on this job. There are three classes of jobs: short (maximum 15 minutes CPU time), medium (maximum one hour CPU time) and long (no CPU time limit). Always use "class = long" for Sopale runs. #@ queue Puts your job in the queue. ./SOPALE1.x This is the command line that actually invokes the SOPALE program. It's the same as what you would type at the shell prompt. To run SOPALE using the Loadleveler, you submit your batch file by typing: llsubmit jobfile.cmd where jobfile.cmd is the name of our command file. A More Complicated Command File ------------------------------- Running SOPALE can be a little convoluted because of the disk space situation on the SP. There usually isn't room enough in your home directory for the large output files generated by SOPALE. However, each node of the SP has a local disk, to be used as temporary storage space, which is accessed as a directory called /usertmp. We can write our output files to the /usertmp of the node where Sopale is running, instead of writing them to our home directory. Our Loadleveler command file creates a directory on the /usertmp disk of whatever node it's running on, copies the input files and executables there, and then runs in that directory. So your output files will end up in a /usertmp directory of one of the nodes. After Sopale has finished running, you must go to that /usertmp directory to find your output. The /usertmp directories are accessible from node 1 (which is where you log in). The /usertmp for node 3 is found at /node/03/usertmp, the /usertmp for node 5 at /node/05/usertmp, and so on. Here is an example of a batch file that we might use to run SOPALE models, for the user 'gstockma' where the model run is named 'n17-8': #!/bin/ksh #@ output = /usertmp/gstockma/n17-8/so.xout #@ error = /usertmp/gstockma/n17-8/so.xerr #@ notification = Always #@ job_type = serial #@ class = long #@ min_processors = 1 #@ max_processors = 1 #@ queue # Make directory on /usertmp for the run JOBDIR=n17-8 if [[ -d /usertmp ]] then cd /usertmp cd gstockma if [[ ! -d $JOBDIR ]] then mkdir $JOBDIR fi cd $JOBDIR cp -r /home/gstockma/sopale/$JOBDIR/* . cp /home/gstockma/sopale/matrices/umat801.41 . ./SOPALE1.x fi Here is a line-by-line description of the command file: #!/bin/ksh This is a shell script command which means "use the shell /bin/ksh to execute these commands" #@ output = /usertmp/gstockma/n17-8/so.xout This Loadleveller option sends the standard output to the file '/usertmp/gstockma/n17-8/so.xout', in other words, the standard output will go to a file called 'so.xout' in the directory '/usertmp/gstockma/n17-8'. #@ error = /usertmp/gstockma/n17-8/so.xerr This Loadleveller option sends the standard error to the file '/usertmp/gstockma/n17-8/so.xerr', in other words, any messages written to standard error will go to a file called 'so.xerr' in the directory '/usertmp/gstockma/n17-8'. #@ notification = Always A Loadleveler option which sends you email when your job starts and when it finishes. The email will be sent to your SP2 account, but you can set your SP2 email to be forwarded to another account. #@ job_type = serial This tells the Loadleveler that the job is serial, not parallel. #@ class = long This tells the Loadleveler that you don't want to impose any CPU time limit on this job. There are several classes of jobs: short (maximum 15 minutes CPU time), medium (maximum 2 hours CPU time), long (maximum 24 hours CPU time), and extra-long (maximum 48 hours CPU time). #@ queue Puts your job in the queue. # Make directory on /usertmp for the run This is a comment line. Any line beginning with "# " is ignored and can be used for useful comments. JOBDIR=n17-8 This sets the variable "JOBDIR" to the value "n17-8". The variable is used later on in this command file. if [[ -d /usertmp ]] then This means "if /usertmp exists and it is a directory, then..." cd /usertmp Change the current directory to /usertmp. cd gstockma change the current directory to gstockam. if [[ ! -d $JOBDIR ]] then mkdir $JOBDIR fi If the directory referred to in JOBDIR (in this case, n17-8) does not exist, then create the directory. cd $JOBDIR Change the current directory to the JOBDIR (in this case, JOBDIR was set to n17-8). cp -r /home/gstockma/sopale/$JOBDIR/* . In this case, $JOBDIR would evaluate to 'n17-8', so this command would copy the directory /home/gstockma/sopale/n17-8 (including files and subdirectories) to the current directory on /usertmp. [The directory /home/gstockma/sopale/n17-8 would be your model directory, which you would have set up before submitting the command file.] cp /home/gstockma/sopale/matrices/umat801.41 . This copies the matrix file to the current directory on /usertmp. ./SOPALE1.x This runs the SOPALE model. fi End of the "if /usertmp exists and it is a directory, then..." ========================== MAKING CHANGES TO THE CODE ========================== If you want to change the size of the grid, you will have to * edit parameters in SOPALE1_i * make a new matrix file * possibly modify and recompile the code The third item will be required if you make the grid much larger and need to allocate more memory in the solver. In the file SOPALE1.h, you will need to change the parameters mnx1 mny1 NMAX ANZMAX LNZMAX TMPMAX mnxplot mnyplot mnx2 mny2