I just copied Objective-C.gitignore. It seemed to do the job with a small Swift project.
The old pattern was too broad and was making VIM ignore Scheme (*.scm) files. Now it only matches files that start with . or _.