Revision history [back]

Edit: The problem is that if you pass a list of OUT files to instantiate one single chnf object, it's gonna explode. It won't explode if you loop through the OUT files one by one. Python's garbage collection should make sure that the unused chnf object would be erased, so perforamce and memory should not be impacted.

You can use multiprocessing to speed up things, here is an example:

import os
import time
import multiprocessing as mp

# you may need to add the PSSBIN path to your environmental variable
import dyntools

def boolOut2Xlsx(str_path_out, str_path_xlsx, bool_show=False,
                 bool_overwritesheet=False, str_sheet='', list_chn=[]):
    '''This function converts the PSSE OUT file into an Excel workbook (XLSX).

    Dependency
    ----------
    dyntools : PSSE Python module

    Parameters
    ----------
    str_path_out : str
        The path of the OUT file.

    str_path_xlsx : str
        The path of the XLSX file.

    bool_overwritesheet : boolean
        Whether to overwrite the current worksheet.

        True = overwrite;
        False = not overwirte.

        Defualt = Flase

    str_sheet : str
        Name of the worksheet.

    list_chn : list
        A list containing the desired channels to be included in the XLSX.

        Empty list means all channels. Note that this could make the XLSX very large.

        Default = emtpy list (all channels)

    Returns
    -------
    list of two bools : list
        This list has two elements. If no exception (conversion assumed successful), the 
        first element will be True, else, it will be False. The second element is always
        True. This is to denote function end (end of conversion).'''

    try:

        # use the CHNF in dyntools, let it read the OUT file path
        obj_chnf = dyntools.CHNF(str_path_out) 

        # if not empty list
        if list_chn:

            list_chn = list_chn

        else:

            # the channel ids are return as a tuple
            tup_id = obj_chnf.get_id(str_path_out)

            # get the total id number
            # turn the tuple into a list then get the length of the list
            int_id_num = len(list(tup_id[1]))

            list_chn   = range(1, int_id_num, 1)

        # this is output line
        obj_chnf.xlsout(channels=list_chn, show=bool_show,
                        xlsfile=str_path_xlsx, outfile=str_path_out,
                        sheet=str_sheet, overwritesheet=bool_overwritesheet)

        return [True, True]

    except Exception as e:

        print(repr(e))

        return [False, True]

def out2xlsx_with_mp(chunk_outfile, chunk_xlsx):

    start = time.time()

    jop = []

    for k, v in zip(chunk_out, chunk_xlsx):

        jop = []

        for z in xrange(0, len(k)):

            p = mp.Process(target=boolOut2Xlsx, args=(k[z], v[z]))

            jop.append(p)

            p.start()

        for j in jop:

            j.join()

    dur = time.time() - start

    print('Time taken with multiprocessing : ' + str(dur))


def out2xlsx_no_mp(list_path_outfile, list_path_xlsx):

    start = time.time()

    for i, j in zip(list_path_outfile, list_path_xlsx):

        boolOut2Xlsx(i, j)

    dur = time.time() - start

    print('Time taken with serial processing : ' + str(dur))


if __name__ == '__main__':

    str_data_dir = <path to your OUT files>

    str_xlsx_dir_mp = <path to where you want to store the XLSX files from multiprocessing>

    str_xlsx_dir_ser = <path to where you want to store the XLSX files from serial processing>

    # you may need to play with this number to get the best performance
    int_chunt_size = 50

    list_filename_outfile = os.listdir(str_data_dir)

    list_filename_outfile = [os.path.join(str_data_dir, i)
                             for i in list_filename_outfile]

    list_filename_xlsx_mp = [os.path.join(str_xlsx_dir_mp,
                                          os.path.basename(i)+'.xlsx')
                             for i in list_filename_outfile]

    list_filename_xlsx_ser = [os.path.join(str_xlsx_dir_ser,
                              os.path.basename(i)+'.xlsx')
                              for i in list_filename_outfile]

    chunk_out  = [list_filename_outfile[x:x+int_chunt_size]
                  for x in xrange(0,
                                  len(list_filename_outfile),
                                  int_chunt_size)]

    chunk_xlsx = [list_filename_xlsx_mp[x:x+int_chunt_size]
                  for x in xrange(0,
                                  len(list_filename_xlsx_mp),
                                  int_chunt_size)]

    print(time.strftime('%H:%M:%S', time.localtime()))

    out2xlsx_with_mp(chunk_out, chunk_xlsx)

    print(time.strftime('%H:%M:%S', time.localtime()))

    out2xlsx_no_mp(list_filename_outfile, list_filename_xlsx_ser)

I tried converting 200 copies of "brn30053007trip.out" (in PSSE's "EXAMPLE" folder), here is the result:

15:53:48
Time taken with multiprocessing : 221.339999914
15:57:30
Time taken with serial processing : 358.616999865

Note that I closed all other apps and just left the computer to do the work. If you are doing other things while benchmarking, the multiprocessing may actually be slower.

If you just want to plot, you can follow this example and apply multiprocessing to plot. Do remember to close the fig each time otherwise your memory will explode.

By the way, I am not using Pool here because I ran into some funny error with win32com.