This may sound trivial, but there is no better way to avoid good placements than a badly prepared target structure. FlexX depends very much on the positions of hydrogens that are typically not part of the PDB input. Thus, you have to tell FlexX in the RDF, where to put them, especially if there are ambiguities like e.g. for HIS. If you provide a reasonable binding pattern, it will be much easier for FlexX to predict reasonable solutions. More details how to prepare the RDF correctly can be found in another outline.

The only difference for atoms that are part of the binding pocket is that FlexX assigns directed interactions only to these atoms so that only these residues can contribute to the match score. However, all protein atoms (and not only those of the active site) are always taken into account for overlap tests and contribute in any case to clash and contact scores. Thus the pocket definition in the RDF determines only to which atoms interaction geometries are assigned. For this reason the pocket definition should be as accurate as possible and there is no need to include any atoms just to avoid free space. There is no free space anyway. However, oftentimes active sites are defined simply by a sphere around a reference point in space. This approach includes a lot of residues that are actually not involved in binding. Nevertheless, geometries are assigned and this widens the search space and may prevent FlexX from focusing to the correct placements.

For the same reason as mentioned in the last section, you should check carefully the solvent accessible residues at the rim of the pocket. These residues will typically not determine the overall binding mode, but only in some cases support a binding mode that is determined by binding patterns in the deeper part of the pocket, because they can interact in the same way with the solvent.

Directed and may be even charged interactions contribute very much to the match and overall score and guide the placements. However, FlexX does not distinguish between buried interactions and those at the rim at the pocket. Thus, FlexX tends to overestimate directed interactions at the rim of the pocket and the effect is that they "pull" the placements out of the pocket. This can be avoided by removing highly flexible residues that can form strong directed interactions like for example ARG or LYS for the active site definition in the RDF.

FlexX employs an incremental docking algorithm, i.e. the ligand is build up fragment by fragment within the binding pocket, starting with up to four different so-called base fragments. Usually the base fragments can and will be placed all over the pocket. However, if the first fragment is already placed at the rim of the pocket, the complete ligand can grow easily towards the outside of the pocket, which leads to overall bad placements. These annoying placements can be avoided by restricting the area where FlexX starts to place the ligand to the deeper part of the pocket or to known specific sub pockets. Such a constraint forces the ligand into the active site without any pre-assumptions about interactions or atom types.

The deeper part of the pocket can be calculated with the command RECEPTOR/DEEPSITE. The more buried part of the active site is identified as follows: For each surface atom of the active site the number of protein atoms within a radius of 10Å is calculated - this is the number of contacts. For each surface atom the fraction of its own number of contacts and the atom with the largest number of contacts is calculated. The deeper part of the pocket can be determined by user defined cut offs for the fraction or the absolute number of contacts. This deep sub pocket or any other sub pocket that can be given as a separate site file in the RDF can then be used to limit the RECEPTOR/TRIHASH command and by this the base placement to this sub pocket.

Last but not least, if you have access to FlexX-Pharm it is always a good idea to constrain the solutions. In almost all cases you will be able to identify some key interactions within the active site that should be addressed by the ligand. If you are not sure which ones are actually essential or you do not want to be too restrictive, assign all of them to be optional and require only one. This alone will skip all predictions that are not at least to some extend in the active site.

An even more trivial way to pull the placements into the binding pocket is to define simply a spherical constraint that forces only any kind of atom into the deep part of the pocket. (atom type: "SMARTS *", new feature of FlexX-Pharm 2.0) But be careful, use such generic constraints advisedly, because they will cause a lot of internal checks and may need a lot of memory.