Ask Your Question

how do you run multiple dynamic contingency idevs using python

asked 2020-12-03 08:20:49 -0500

Ransome gravatar image

I have created dynamic contingency idevs that run one after the other. Whenever any of them couldn't solve or complete the simulation, the process stop. I have to disabled that idevs and restart the process to move to the next idevs. I'm looking for a way that even if one crashes or couldn't solve, it should move on execute the next contingency idevs and only stop after the last one. that way I can go back and see the unsolved idevs for further analysis.

edit retag flag offensive close merge delete

2 answers

Sort by ┬╗ oldest newest most voted

answered 2020-12-07 10:56:25 -0500

GaryB gravatar image

The way we do it is complicated but bulletproof. In short, use the multiprocessing Module to run the simulations. You will need to put your run script into a function so that you can pass that function to multiprocessing. Please see the code below. The .Process() function starts the simulation by passing the next faultidev to the Function rundynsim(). I have a 28 core machine, so I load up 36 processes and let the multi-threading engine handle the balancing. This is set with the numprocesses Variable. If you want to run sequentially, you can use numprocesses=1.

    while fault_idevs:
    # if we aren't using all the processors AND there is still data left to
    # compute, then spawn another thread
    #Create a new thread if we haven't used all available to us.  Only create 
    #   one per sleep cycle.
    if( len(threads) < num_processes):    
        fault_being_executed = fault_idevs[0]       # Strings passed by value
        p = multiprocessing.Process(target=run_dynsim, args=[cnvcase, snap, fault_idevs.pop(0), lock, generic_VTG])
        print p, p.is_alive()
        # Set us up a dictionary of running processes and some info so that we can 
        #  check to see if they have stalled
        running_processes[] = {'Fault_Name': fault_being_executed, 'Log_Size': 0, 'Out_Size': 0, 'Num_Stalls': 0, 'Start_Time':}
        for thread in threads:                        # Remove any threads that have finished
            if not thread.is_alive():
                del running_processes[]
                print 'REMOVING A THREAD'
    # Let's check on our processes every minute
    stalled_process_check(running_processes, threads, lock, 60, len(fault_idevs), True)

To answer the main part of your question, running them in separate processes allows you to check and see if they are still working. If not, terminate them and move on. Some of mine can take more than 30 minutes to complete.

See the function below. The gist of it is to check the length of the log file and out Files; and if they aren't changing, terminate the process. The os.stat Functions make a call to get file sizes. These get stored and compared with the next stalled process check. This gets run every minute, so hung processes will be terminated within about three minutes. (The allowed number of stalls below is set to one, so it will terminate a minute after detection.)

# =============================================================================
# Function stalled_process_check
# Input: running_processes - the list of known running processes
#        threads - the list of threads being run by multiprocessing
#        lock - file lock for writing a status update to file
#        update_seconds - How often to actually run the main routine in this
#                         function
#        no_of_waiting - number of processes to queue or threads that are 
#                        waiting at the end
#        still_queueing - Status of whether there are more threads
#  Operation: This code loop through running_processes and save .log and 
#             .out file sizes.  If they don't change, it will record a stall.
#             If it hasn't changed the next time either, it will terminate the 
#             thread and let other code do the garbage collection. There is a 
#             status update at the end.
#  Output: Operates ...
edit flag offensive delete link more

answered 2020-12-05 00:28:48 -0500

Carlos G gravatar image

You lose the control once you have initialized the dynamic simulaci├│n, I mean, if you have a python code running several simulations you run the risk to have a crash simulation. The option Is to save every result under *.dat or *.out files. the risk increase under a not well sintonized case.cagm_mxc

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

[hide preview]

Question Tools

1 follower


Asked: 2020-12-03 08:20:49 -0500

Seen: 312 times

Last updated: Dec 07 '20