New York’s new cybersecurity rules for banks, insurance companies, and registered cryptocurrency groups, may leave small and medium-sized financial institutions struggling to comply.

The New York Department of Financial Services rules, which has a Sept. 4 compliance deadline, require companies to keep audit trails for online financial transactions and cybersecurity episodes. Companies also will need to maintain policies and procedures to secure financial apps used on mobile devices.

Many larger players, such as JPMorgan Chase & Co., The Bank of...