The single-writer/multi-reader searchable encryption (SMSE) allows an arbitrary authorized user to submit a valid search token and get the corresponding encrypted identifiers. In order to achieve fine-grained access control, the identifiers are encrypted by the attribute-based encryption. In this case, the user can decrypt a ciphertext only when the access policy in it matches the user’s attribute set. However, the server unable to determine whether the user can decrypt a certain ciphertext without the knowledge of the user’s attribute set. As a result, all the ciphertexts based on a search token have to be returned to the user, which causes unnecessary communication and decryption costs. In this paper, we propose a new SMSE scheme, in which the server just needs to return the ones which can be decrypted by the user rather than the whole search results. In order to achieve this goal, we present a server-side match technique with which the server can test whether the user can decrypt a ciphertext without knowing the user’s attribute set. Furthermore, the decryption computation is very efficient, irrespective of the structure of access policy. Therefore, both the communication and decryption overheads are dramatically reduced in our scheme.