Dyntools: Dealing with many .OUT files when converting to .csv

answered 2019-10-15 06:09:08 -0500

drsgao
76 ●5 ●7 ●11

updated 2019-10-15 06:11:16 -0500

I think this may be a regression problem since I have tested it with v33.4 with 128 OUT files (56MB each) and with "txtout" and got no problem. I don't have a higher version of PSSE so I could not test the "csvout" (Siemens says it starts to be available in v33.12.1). I would suggest you to raise a ticket with Siemens and see what's the official line. Note that I have seen a lot of bugs reported in v34, which is the version Siemens started to support Python 3 (I am not sure the exact version though).

In the mean time, you may be able to work around it using multiprocessing since the memory spaces are separated, and thus you may be able to keep the register in check (I suspect some kind of stack or memory issues though).

Again, I don't have the "csvout" in my version, so the following function is in "txtout". You would need to change it to "csvout" (you might be able to use the TXTs as CSVs if your channel labels are not too long).

# -*- coding: utf-8 -*-

'''
Testing the "dyntools" with multiprocessing to try to work around the
register fill-up issue.

Change Log
----------------------

* **Notable changes:**
   + Version : 1.0.0
      - First version

'''

import os
import glob
import multiprocessing as mp

# you may need to add more code here to be able to import this module
import dyntools


def out2txt(str_path_outfile):
    '''
    Very simple function using "dyntools" to convert an OUT file to an
    TXT file.

    Note that the "txtout" method can take more arguments, this function
    keeps the arguments at the minimum. Change the arguments as your
    needs.

    Parameters
    ------------
    str_path_outfile : str

        The full file path of the OUT file that is to be converted.

    Returns
    -------
    None.
    '''

    obj_chnf = dyntools.CHNF(str_path_outfile)

    obj_chnf.txtout(outfile=str_path_outfile)

    # just an feedback line, you can comment this out
    print(str_path_outfile + ' done!')


if __name__ == '__main__':

    # set your input directory path here, you can also use the "input"
    # function for better interactions
    str_path_dir = r'<your/dir/path>'

    # pattern for glob
    str_pattern = r'*.out'

    list_files = glob.glob(os.path.join(str_path_dir, str_pattern))

    # make a pool with 4 processes
    pool = mp.Pool(4)

    # pool of "out2txt" to deal with the OUT files, note that this will
    # block
    pool.map(out2txt, list_files)

    # some exit code, may be ok without them
    pool.close()

    pool.join()

    print('All Done!')

Tested with v33.4 with 128 OUT files (56MB each). All successful.